[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:12 dist-test-slave-14jq krb5kdc[3040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:12 dist-test-slave-14jq krb5kdc[3040](info): set up 2 sockets
Sep 27 15:19:12 dist-test-slave-14jq krb5kdc[3040](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:19:13.079567 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.588s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:13 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:13.283388 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:43713
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:43713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:13.721141 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:13.721553 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:13.721654 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:13.730578 3053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:13.730770 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:13.730822 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:13.730877 3053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:13.730924 3053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:13.744940 3053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:43713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:43713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:13.746817 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:13.748566 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:13.764618 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.764680 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.764575 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.766242 3053 server_base.cc:1034] running on GCE node
I20260927 15:19:13.767258 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:13.769176 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:13.772589 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790522353772431 us; error 157 us; skew 500 ppm
I20260927 15:19:13.773067 3053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:13 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:13.777020 3053 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:13.778992 3053 webserver.cc:460] Webserver started at http://127.2.246.126:45673/ using document root <none> and password file <none>
I20260927 15:19:13.779894 3053 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:13.780051 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:13.780403 3053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:13.783972 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "164fe798368b4ff59d83e72ce5e11df1"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-14jq"
server_key: "a6b33551ace4602f7c8f984500899ad3"
server_key_iv: "7b6683fc27ae4f603c818d2ce686021c"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.784814 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "164fe798368b4ff59d83e72ce5e11df1"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-14jq"
server_key: "a6b33551ace4602f7c8f984500899ad3"
server_key_iv: "7b6683fc27ae4f603c818d2ce686021c"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.791272 3053 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:19:13.796345 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.798418 3053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:13.798660 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "164fe798368b4ff59d83e72ce5e11df1"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-14jq"
server_key: "a6b33551ace4602f7c8f984500899ad3"
server_key_iv: "7b6683fc27ae4f603c818d2ce686021c"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.798877 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:13.820006 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:13.825088 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:13.825414 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:13.849408 3053 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:43713
I20260927 15:19:13.849511 3119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:43713 every 8 connection(s)
I20260927 15:19:13.850991 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:13.856374 3120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:13.856927 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3053
I20260927 15:19:13.857340 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:13.860265 3033 external_mini_cluster.cc:1428] Setting key 8c991f7b86ce4a0556a5b26f2aa3b0f9
I20260927 15:19:13.874334 3120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1: Bootstrap starting.
I20260927 15:19:13.880508 3120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:13.882486 3120 log.cc:824] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:13.886807 3120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1: No bootstrap required, opened a new log
Sep 27 15:19:13 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:13.896541 3120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } }
I20260927 15:19:13.897063 3120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:13.897176 3120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164fe798368b4ff59d83e72ce5e11df1, State: Initialized, Role: FOLLOWER
I20260927 15:19:13.898075 3120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } }
I20260927 15:19:13.898403 3120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:13.898582 3120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:13.898783 3120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:13.902035 3120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } }
I20260927 15:19:13.902812 3120 leader_election.cc:302] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164fe798368b4ff59d83e72ce5e11df1; no voters:
I20260927 15:19:13.903918 3120 leader_election.cc:288] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:13.904139 3125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:13.905434 3125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [term 1 LEADER]: Becoming Leader. State: Replica: 164fe798368b4ff59d83e72ce5e11df1, State: Running, Role: LEADER
I20260927 15:19:13.906754 3125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } }
I20260927 15:19:13.908638 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.867375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.2.246.126:43713)
0927 15:19:13.868831 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.868855 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.870230 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 15:19:13.881639 (+ 11409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.881685 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.882507 (+ 822us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.883151 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:13.887175 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.888599 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.890453 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.891188 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.891407 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.897748 (+ 6341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:13.897758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:13.897775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:13.897825 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:13.901520 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.902467 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.902474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.902480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.902693 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.903219 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.903225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.903228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.904002 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:13.904454 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.906327 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.907278 (+ 951us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":543,"threads_started":1}
I20260927 15:19:13.909759 3120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:13.912011 3126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164fe798368b4ff59d83e72ce5e11df1. Latest consensus state: current_term: 1 leader_uuid: "164fe798368b4ff59d83e72ce5e11df1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } } }
I20260927 15:19:13.912546 3126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:13.912197 3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164fe798368b4ff59d83e72ce5e11df1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164fe798368b4ff59d83e72ce5e11df1" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43713 } } }
I20260927 15:19:13.912694 3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:13.918026 3130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:13.927632 3130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:13.942574 3130 catalog_manager.cc:1347] Generated new cluster ID: 52831c09d32b4a89a0289d76f7ceee15
I20260927 15:19:13.942770 3130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:13.976761 3130 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:13.978610 3130 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:13.993929 3130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164fe798368b4ff59d83e72ce5e11df1: Generated new TSK 0
I20260927 15:19:13.995302 3130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:14.063172 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:14.536216 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:14.536674 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:14.536873 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.546478 3146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.546693 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.546914 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:14.559391 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:14.561293 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.563346 3146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.581348 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.581426 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.581683 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.582890 3146 server_base.cc:1034] running on GCE node
I20260927 15:19:14.583741 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.585074 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.586374 3146 hybrid_clock.cc:648] HybridClock initialized: now 1790522354586290 us; error 72 us; skew 500 ppm
I20260927 15:19:14.586835 3146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:14 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.592667 3146 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:14.594748 3146 webserver.cc:460] Webserver started at http://127.2.246.65:36183/ using document root <none> and password file <none>
I20260927 15:19:14.595598 3146 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.595789 3146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.596216 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:14.599742 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "48621479c7214e4080d4e1b3890ab87d"
format_stamp: "Formatted at 2026-09-27 15:19:14 on dist-test-slave-14jq"
server_key: "3cfff32d0c2de61361e08f968a4db51e"
server_key_iv: "3098f1894792ec027036221e8a11754e"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.600594 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "48621479c7214e4080d4e1b3890ab87d"
format_stamp: "Formatted at 2026-09-27 15:19:14 on dist-test-slave-14jq"
server_key: "3cfff32d0c2de61361e08f968a4db51e"
server_key_iv: "3098f1894792ec027036221e8a11754e"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.607324 3146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:19:14.612175 3161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.614327 3146 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 15:19:14.614562 3146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "48621479c7214e4080d4e1b3890ab87d"
format_stamp: "Formatted at 2026-09-27 15:19:14 on dist-test-slave-14jq"
server_key: "3cfff32d0c2de61361e08f968a4db51e"
server_key_iv: "3098f1894792ec027036221e8a11754e"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.614820 3146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:14.632294 3146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:14.638252 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:14.638609 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:14.639878 3146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:14.641505 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:14.641599 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.641690 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:14.641785 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.685962 3146 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:46855
I20260927 15:19:14.686133 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:46855 every 8 connection(s)
I20260927 15:19:14.687402 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:14.691309 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3146
I20260927 15:19:14.691599 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:14.692299 3033 external_mini_cluster.cc:1428] Setting key 16d5d9072607cc394bcaa5bca0679f34
Sep 27 15:19:14 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:14.718389 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.691280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:51153 (local address 127.2.246.126:43713)
0927 15:19:14.693998 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.694006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.694030 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:19:14.697808 (+ 3778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.697815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.697968 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.698104 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:14.700665 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.701752 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.704809 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.705242 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.705378 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.711036 (+ 5658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:14.711046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:14.711050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:14.711088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:14.714085 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.715255 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.715260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.715263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.715435 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.716051 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.716056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.716058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.716426 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:14.716571 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.717664 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.717995 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2580,"thread_start_us":2570,"threads_started":1}
I20260927 15:19:14.719750 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.692046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43713 (local address 127.2.246.65:51153)
0927 15:19:14.693513 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.693610 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.697400 (+ 3790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.698436 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.698448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.699299 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:14.700378 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.700413 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.702017 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.702026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.704582 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.704609 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.705579 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.705591 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.706052 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.707058 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:14.707106 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:14.710696 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:14.714355 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.714367 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.714389 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.715054 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.715606 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.715614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.715621 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.715905 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.716604 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:14.716616 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:14.717182 (+ 566us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.717542 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.718246 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":571,"threads_started":1}
I20260927 15:19:14.721807 3274 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43713
I20260927 15:19:14.722470 3274 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:14.723654 3274 heartbeater.cc:511] Master 127.2.246.126:43713 requested a full tablet report, sending...
I20260927 15:19:14.726537 3085 ts_manager.cc:194] Registered new tserver with Master: 48621479c7214e4080d4e1b3890ab87d (127.2.246.65:46855)
I20260927 15:19:14.728223 3085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:51153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:14.743017 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:15.171257 3280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:15.171617 3280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:15.171797 3280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:15.179929 3280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:15.180087 3280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:15.180274 3280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:15.190742 3280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:15.192229 3280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:15.193759 3280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:15.208523 3285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.208585 3286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.208601 3288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:15.210300 3280 server_base.cc:1034] running on GCE node
I20260927 15:19:15.211001 3280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:15.212088 3280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:15.213312 3280 hybrid_clock.cc:648] HybridClock initialized: now 1790522355213244 us; error 61 us; skew 500 ppm
I20260927 15:19:15.213634 3280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:15 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:15.217101 3280 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:15.218847 3280 webserver.cc:460] Webserver started at http://127.2.246.66:41863/ using document root <none> and password file <none>
I20260927 15:19:15.219614 3280 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:15.219767 3280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:15.220095 3280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:15.223124 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "f75653f9fbff4d83abae751841f30e34"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "0734bc59d9703f08c7ff43a601ea83ad"
server_key_iv: "b4d2029f2a1fa3e2ecae18995c49299e"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.223923 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "f75653f9fbff4d83abae751841f30e34"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "0734bc59d9703f08c7ff43a601ea83ad"
server_key_iv: "b4d2029f2a1fa3e2ecae18995c49299e"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.229965 3280 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:19:15.234215 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.235913 3280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:15.236186 3280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "f75653f9fbff4d83abae751841f30e34"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "0734bc59d9703f08c7ff43a601ea83ad"
server_key_iv: "b4d2029f2a1fa3e2ecae18995c49299e"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.236806 3280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:15.264137 3280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:15.269160 3280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:15.269486 3280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:15.270609 3280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:15.272188 3280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:15.272294 3280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.272452 3280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:15.272562 3280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.315900 3280 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:33175
I20260927 15:19:15.315955 3407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:33175 every 8 connection(s)
I20260927 15:19:15.317516 3280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:15.326480 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3280
I20260927 15:19:15.326730 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:15.327302 3033 external_mini_cluster.cc:1428] Setting key 2d1e9673f35a1522edd5698c2bc0a987
Sep 27 15:19:15 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:15.346077 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.321128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36013 (local address 127.2.246.126:43713)
0927 15:19:15.321762 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:15.321771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:15.323190 (+ 1419us) server_negotiation.cc:407] Connection header received
0927 15:19:15.325157 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:15.325162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:15.325265 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:15.325390 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:15.327315 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:15.328151 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:15.330689 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:15.331346 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:15.331498 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:15.338757 (+ 7259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:15.338763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:15.338767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:15.338811 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:15.342408 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:15.343389 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:15.343396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:15.343400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:15.343543 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:15.344029 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:15.344035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:15.344039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:15.344352 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:15.344519 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:15.345374 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 15:19:15.345630 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
I20260927 15:19:15.347281 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.322021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43713 (local address 127.2.246.66:36013)
0927 15:19:15.322912 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.322975 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:15.324848 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:15.325644 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:15.325653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:15.326375 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:15.327134 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:15.327155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:15.328416 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:15.328423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:15.330438 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:15.330464 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:15.331701 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:15.331713 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:15.332203 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:15.333247 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:15.333290 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:15.338347 (+ 5057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:15.342604 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:15.342618 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:15.342643 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:15.343239 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:15.343672 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:15.343678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:15.343682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:15.343897 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:15.344490 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:15.344500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:15.344897 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 15:19:15.345261 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 15:19:15.345943 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":353,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:15.349195 3408 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43713
I20260927 15:19:15.349756 3408 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:15.350885 3408 heartbeater.cc:511] Master 127.2.246.126:43713 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:19:15.353253 3085 ts_manager.cc:194] Registered new tserver with Master: f75653f9fbff4d83abae751841f30e34 (127.2.246.66:33175)
I20260927 15:19:15.354329 3085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:36013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:15.378257 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:15.733772 3274 heartbeater.cc:503] Master 127.2.246.126:43713 was elected leader, sending a full tablet report...
W20260927 15:19:15.832994 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:15.833421 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:15.833627 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:15.842847 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:15.843057 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:15.843314 3414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:15.856093 3414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:15.858709 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:15.860558 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:15.879381 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.879704 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.880285 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:15.880734 3414 server_base.cc:1034] running on GCE node
I20260927 15:19:15.881565 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:15.882961 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:15.884258 3414 hybrid_clock.cc:648] HybridClock initialized: now 1790522355884200 us; error 47 us; skew 500 ppm
I20260927 15:19:15.884785 3414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:15 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:15.889556 3414 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:15.891927 3414 webserver.cc:460] Webserver started at http://127.2.246.67:45169/ using document root <none> and password file <none>
I20260927 15:19:15.892889 3414 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:15.893072 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:15.893472 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:15.897651 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "9df705d4bfb64620844bf66c3307b872"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "da118aaa7ded5d312ef43ad2912846b0"
server_key_iv: "a2315705124f2442900ab814782408db"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.899888 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "9df705d4bfb64620844bf66c3307b872"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "da118aaa7ded5d312ef43ad2912846b0"
server_key_iv: "a2315705124f2442900ab814782408db"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.908052 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20260927 15:19:15.912989 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.915071 3414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:15.915671 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "9df705d4bfb64620844bf66c3307b872"
format_stamp: "Formatted at 2026-09-27 15:19:15 on dist-test-slave-14jq"
server_key: "da118aaa7ded5d312ef43ad2912846b0"
server_key_iv: "a2315705124f2442900ab814782408db"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.916013 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:15.965207 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:15.971251 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:15.971658 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:15.972968 3414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:15.974959 3414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:15.975083 3414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.975260 3414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:15.975340 3414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:16.023559 3414 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:36215
I20260927 15:19:16.023608 3541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:36215 every 8 connection(s)
I20260927 15:19:16.025449 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:16.032557 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3414
I20260927 15:19:16.032806 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:16.033453 3033 external_mini_cluster.cc:1428] Setting key f03ba08057c7771b04de10f8bb026c9a
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:16.053368 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.028956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:55297 (local address 127.2.246.126:43713)
0927 15:19:16.029643 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.029651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.031643 (+ 1992us) server_negotiation.cc:407] Connection header received
0927 15:19:16.034815 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.034824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.034965 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.035132 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.037571 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.038912 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.040930 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.041412 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.041541 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.045422 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.045428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.045432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.045521 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.049149 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.050204 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.050214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.050219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.050397 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.051002 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.051007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.051011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.051509 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.051693 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.052759 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.053003 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260927 15:19:16.054572 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.029825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43713 (local address 127.2.246.67:55297)
0927 15:19:16.031372 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.031446 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:16.034305 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:16.035513 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:16.035523 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:16.036346 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:16.037346 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.037378 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.039137 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.039143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.040751 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.040765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.041787 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.041792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.042077 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.042861 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:16.042888 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:16.045118 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:16.049352 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.049360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.049376 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.050047 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.050560 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.050564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.050566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.050851 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.051695 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:16.051705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:16.052247 (+ 542us) client_negotiation.cc:769] Sending connection context
0927 15:19:16.052642 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 15:19:16.053151 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":774,"threads_started":1}
I20260927 15:19:16.056337 3542 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43713
I20260927 15:19:16.056649 3542 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:16.057451 3542 heartbeater.cc:511] Master 127.2.246.126:43713 requested a full tablet report, sending...
I20260927 15:19:16.059288 3085 ts_manager.cc:194] Registered new tserver with Master: 9df705d4bfb64620844bf66c3307b872 (127.2.246.67:36215)
I20260927 15:19:16.060390 3085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:55297
I20260927 15:19:16.066222 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:16.121068 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.101395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.2.246.65:46855)
0927 15:19:16.102309 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.102317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.102370 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:19:16.102436 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.102443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.102745 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.102887 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.104788 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.106125 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.107903 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.108509 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.108639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.114505 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.114521 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.114539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.114579 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.117653 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.118678 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.118688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.118691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.118823 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.119303 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.119306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.119308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.119719 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.119944 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.120391 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.120635 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":549,"threads_started":1}
W20260927 15:19:16.122970 3189 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47880
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:16.176949 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.159119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.2.246.65:46855)
0927 15:19:16.159550 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.159559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.159804 (+ 245us) server_negotiation.cc:407] Connection header received
0927 15:19:16.160035 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.160040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.160135 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.160255 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.161759 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.162668 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.164026 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.164499 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.164618 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.169162 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.169170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.169174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.169229 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.172678 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.173631 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.173637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.173640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.174241 (+ 601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.175187 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.175192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.175194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.175599 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.175827 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.176245 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.176563 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 15:19:16.179191 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3146
I20260927 15:19:16.194977 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.180458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.2.246.65:46855)
0927 15:19:16.180735 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.180742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.181199 (+ 457us) server_negotiation.cc:407] Connection header received
0927 15:19:16.181428 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.181433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.181517 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.181638 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.183103 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.184042 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.185506 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.186074 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.186206 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.188651 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.188657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.188662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.188704 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.192040 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.192759 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.192764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.192767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.192953 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.193334 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.193348 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.193350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.193666 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.193973 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.194255 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.194532 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:19:16.195993 3269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:16.296084 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3146
I20260927 15:19:16.313746 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3280
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:16.340451 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.314922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.2.246.66:33175)
0927 15:19:16.315783 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.315795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.315832 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:19:16.315937 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.315947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.316280 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.316495 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.317714 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.319579 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.321304 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.322322 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.322467 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.333184 (+ 10717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.333208 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.333230 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.333288 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.336699 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.337640 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.337650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.337653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.337911 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.338487 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.338492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.338495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.338939 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.339206 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.339684 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.339945 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":514,"threads_started":1}
I20260927 15:19:16.342357 3403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:16.430073 3408 heartbeater.cc:503] Master 127.2.246.126:43713 was elected leader, sending a full tablet report...
I20260927 15:19:16.431448 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3280
W20260927 15:19:16.456316 3072 connection.cc:535] server connection from 127.2.246.66:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:19:16.456799 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3414
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:16.475224 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.457955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.2.246.67:36215)
0927 15:19:16.458543 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.458551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.458583 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:19:16.458654 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.458661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.458897 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.459094 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.460517 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.461581 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.463052 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.463669 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.463816 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.468575 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.468587 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.468603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.468650 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.471897 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.472814 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.472821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.472824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.472965 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.473387 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.473397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.473399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.473779 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.474190 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.474535 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.474850 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":263,"threads_started":1}
I20260927 15:19:16.476995 3537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:16.574218 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3414
I20260927 15:19:16.593050 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3053
I20260927 15:19:16.594740 3115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:16.715566 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3053
2026-09-27T15:19:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5286 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3564](info): set up 2 sockets
Sep 27 15:19:16 dist-test-slave-14jq krb5kdc[3564](info): commencing operation
krb5kdc: starting...
W20260927 15:19:17.874272 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:18 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:18.086885 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42615
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:18.440897 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:18.441216 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:18.441349 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:18.448926 3577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:18.449051 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:18.449100 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:18.449215 3577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:18.449286 3577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:18.459653 3577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:18.461077 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:18.462424 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:18.475732 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:18.475769 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:18.475726 3585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:18.477068 3577 server_base.cc:1034] running on GCE node
I20260927 15:19:18.478051 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:18.479686 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:18.480913 3577 hybrid_clock.cc:648] HybridClock initialized: now 1790522358480856 us; error 64 us; skew 500 ppm
I20260927 15:19:18.481325 3577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:18 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:18.485002 3577 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:18.486871 3577 webserver.cc:460] Webserver started at http://127.2.246.126:45803/ using document root <none> and password file <none>
I20260927 15:19:18.487612 3577 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:18.487762 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:18.488085 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:18.491178 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "082e816e1f824c3fbbc0cbd094788a3d"
format_stamp: "Formatted at 2026-09-27 15:19:18 on dist-test-slave-14jq"
server_key: "0d15fb1f9484f91d238b57b24af5b105"
server_key_iv: "68331fc22ff3495f10396d73bbce8321"
server_key_version: "encryptionkey@0"
I20260927 15:19:18.491935 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "082e816e1f824c3fbbc0cbd094788a3d"
format_stamp: "Formatted at 2026-09-27 15:19:18 on dist-test-slave-14jq"
server_key: "0d15fb1f9484f91d238b57b24af5b105"
server_key_iv: "68331fc22ff3495f10396d73bbce8321"
server_key_version: "encryptionkey@0"
I20260927 15:19:18.497725 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:19:18.502296 3592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:18.503975 3577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:18.504204 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "082e816e1f824c3fbbc0cbd094788a3d"
format_stamp: "Formatted at 2026-09-27 15:19:18 on dist-test-slave-14jq"
server_key: "0d15fb1f9484f91d238b57b24af5b105"
server_key_iv: "68331fc22ff3495f10396d73bbce8321"
server_key_version: "encryptionkey@0"
I20260927 15:19:18.504429 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:18.521255 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:18.526733 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:18.527055 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:18.549902 3577 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:42615
I20260927 15:19:18.550016 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:42615 every 8 connection(s)
I20260927 15:19:18.551492 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:18.556403 3644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:18.559530 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3577
I20260927 15:19:18.559809 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:18.560657 3033 external_mini_cluster.cc:1428] Setting key 273fd135beaed33709a17d9860df9b2f
I20260927 15:19:18.571610 3644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d: Bootstrap starting.
I20260927 15:19:18.576517 3644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d: Neither blocks nor log segments found. Creating new log.
Sep 27 15:19:18 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:18.578446 3644 log.cc:824] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:18.583571 3644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d: No bootstrap required, opened a new log
I20260927 15:19:18.589314 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:18.563724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.2.246.126:42615)
0927 15:19:18.564796 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:18.564818 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:18.564874 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:19:18.566964 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:18.566994 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:18.567492 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:18.568004 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:18.569534 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:18.570975 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:18.572336 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:18.573102 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:18.573322 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:18.578047 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:18.578074 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:18.578108 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:18.578192 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:18.583032 (+ 4840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:18.584132 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:18.584146 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:18.584165 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:18.584434 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:18.584860 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:18.584866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:18.584869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:18.585796 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:18.586208 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:18.586738 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 15:19:18.587387 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":471,"threads_started":1}
I20260927 15:19:18.595711 3644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } }
I20260927 15:19:18.596199 3644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:18.596285 3644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082e816e1f824c3fbbc0cbd094788a3d, State: Initialized, Role: FOLLOWER
I20260927 15:19:18.596971 3644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } }
I20260927 15:19:18.597261 3644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:18.597419 3644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:18.597646 3644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:18.602712 3644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } }
I20260927 15:19:18.603552 3644 leader_election.cc:302] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082e816e1f824c3fbbc0cbd094788a3d; no voters:
I20260927 15:19:18.604493 3644 leader_election.cc:288] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:18.604696 3649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:18.605767 3649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [term 1 LEADER]: Becoming Leader. State: Replica: 082e816e1f824c3fbbc0cbd094788a3d, State: Running, Role: LEADER
I20260927 15:19:18.606590 3649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } }
I20260927 15:19:18.607568 3644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:18.610247 3650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082e816e1f824c3fbbc0cbd094788a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } } }
I20260927 15:19:18.611361 3650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:18.610870 3651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082e816e1f824c3fbbc0cbd094788a3d. Latest consensus state: current_term: 1 leader_uuid: "082e816e1f824c3fbbc0cbd094788a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082e816e1f824c3fbbc0cbd094788a3d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42615 } } }
I20260927 15:19:18.612097 3651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:18.612306 3655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:18.618953 3655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:18.630110 3655 catalog_manager.cc:1347] Generated new cluster ID: 588fcb4bd0264309990111c80fadd654
I20260927 15:19:18.630296 3655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:18.671109 3655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:18.673318 3655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:18.683422 3655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082e816e1f824c3fbbc0cbd094788a3d: Generated new TSK 0
I20260927 15:19:18.684543 3655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:18.738497 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:19.175112 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:19.175513 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:19.175693 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:19.190337 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:19.190551 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:19.190815 3670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:19.210904 3670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:19.212558 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:19.214407 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:19.234575 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:19.234823 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:19.236331 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:19.236800 3670 server_base.cc:1034] running on GCE node
I20260927 15:19:19.237632 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:19.238844 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:19.242277 3670 hybrid_clock.cc:648] HybridClock initialized: now 1790522359242190 us; error 80 us; skew 500 ppm
I20260927 15:19:19.242743 3670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:19 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:19.247109 3670 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:19.249114 3670 webserver.cc:460] Webserver started at http://127.2.246.65:37391/ using document root <none> and password file <none>
I20260927 15:19:19.250082 3670 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:19.250248 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:19.250605 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:19.254065 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "a09c8b41b41e429891f95c640133e21e"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "9ee3a4dd31d1dbd4feb4bcb1a4c62a40"
server_key_iv: "f7e119be4e7ce3e9eea4e9580cb0209f"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.254972 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "a09c8b41b41e429891f95c640133e21e"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "9ee3a4dd31d1dbd4feb4bcb1a4c62a40"
server_key_iv: "f7e119be4e7ce3e9eea4e9580cb0209f"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.261758 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:19:19.266616 3685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:19.268747 3670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:19.269012 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "a09c8b41b41e429891f95c640133e21e"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "9ee3a4dd31d1dbd4feb4bcb1a4c62a40"
server_key_iv: "f7e119be4e7ce3e9eea4e9580cb0209f"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.269250 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:19.291604 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:19.296720 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:19.297158 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:19.298563 3670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:19.300333 3670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:19.300452 3670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:19.300626 3670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:19.300702 3670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:19.345513 3670 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:43161
I20260927 15:19:19.345626 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:43161 every 8 connection(s)
I20260927 15:19:19.347092 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:19.350208 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3670
I20260927 15:19:19.350492 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:19.351598 3033 external_mini_cluster.cc:1428] Setting key b4c98ef71bfbf1fed49e969b8eec006a
Sep 27 15:19:19 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:19.378922 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:19.351123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:37241 (local address 127.2.246.126:42615)
0927 15:19:19.351866 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:19.351874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:19.353316 (+ 1442us) server_negotiation.cc:407] Connection header received
0927 15:19:19.356408 (+ 3092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:19.356413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:19.356507 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:19.356661 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:19.360238 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:19.361184 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:19.365210 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:19.365706 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:19.365959 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:19.372092 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:19.372100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:19.372103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:19.372145 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:19.375283 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:19.376233 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:19.376239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:19.376242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:19.376371 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:19.376845 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:19.376855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:19.376857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:19.377213 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:19.377386 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:19.378276 (+ 890us) server_negotiation.cc:299] Negotiation successful
0927 15:19:19.378527 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":334,"threads_started":1}
I20260927 15:19:19.380075 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:19.351948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42615 (local address 127.2.246.65:37241)
0927 15:19:19.353077 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:19.353142 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:19.355909 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:19.357107 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:19.357126 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:19.358151 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:19.359841 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:19.359888 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:19.361602 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:19.361610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:19.364817 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:19.364857 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:19.366185 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:19.366197 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:19.366624 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:19.367798 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:19.367850 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:19.371718 (+ 3868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:19.375492 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:19.375507 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:19.375529 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:19.376070 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:19.376498 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:19.376503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:19.376507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:19.376714 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:19.377359 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:19.377368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:19.377776 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 15:19:19.378162 (+ 386us) client_negotiation.cc:240] Negotiation successful
0927 15:19:19.378778 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":467,"threads_started":1}
I20260927 15:19:19.382335 3798 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42615
I20260927 15:19:19.382959 3798 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:19.384239 3798 heartbeater.cc:511] Master 127.2.246.126:42615 requested a full tablet report, sending...
I20260927 15:19:19.387512 3609 ts_manager.cc:194] Registered new tserver with Master: a09c8b41b41e429891f95c640133e21e (127.2.246.65:43161)
I20260927 15:19:19.389081 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:37241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:19.410151 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:19.856087 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:19.856498 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:19.856612 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:19.866207 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:19.866381 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:19.866528 3804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:19.878216 3804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:19.879916 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:19.881493 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:19.897425 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:19.897541 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:19.897541 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:19.898947 3804 server_base.cc:1034] running on GCE node
I20260927 15:19:19.899801 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:19.901093 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:19.902390 3804 hybrid_clock.cc:648] HybridClock initialized: now 1790522359902309 us; error 61 us; skew 500 ppm
I20260927 15:19:19.902832 3804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:19 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:19.906883 3804 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:19.909195 3804 webserver.cc:460] Webserver started at http://127.2.246.66:43299/ using document root <none> and password file <none>
I20260927 15:19:19.910231 3804 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:19.910403 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:19.910774 3804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:19.914958 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "04414fdb7a0d48debcd1b6d963ab190d"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "317a6465628aac5f70135c8396d13ccf"
server_key_iv: "2384d86b93ee0695c750aee7bd7c288c"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.915957 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "04414fdb7a0d48debcd1b6d963ab190d"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "317a6465628aac5f70135c8396d13ccf"
server_key_iv: "2384d86b93ee0695c750aee7bd7c288c"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.923171 3804 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:19:19.928390 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:19.931032 3804 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:19:19.931362 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "04414fdb7a0d48debcd1b6d963ab190d"
format_stamp: "Formatted at 2026-09-27 15:19:19 on dist-test-slave-14jq"
server_key: "317a6465628aac5f70135c8396d13ccf"
server_key_iv: "2384d86b93ee0695c750aee7bd7c288c"
server_key_version: "encryptionkey@0"
I20260927 15:19:19.931706 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:19.964526 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:19.970327 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:19.970701 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:19.972045 3804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:19.974112 3804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:19.974256 3804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:19:19.974445 3804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:19.974529 3804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:20.017218 3804 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:32885
I20260927 15:19:20.017325 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:32885 every 8 connection(s)
I20260927 15:19:20.019009 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:20.027213 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3804
I20260927 15:19:20.027510 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:20.027984 3033 external_mini_cluster.cc:1428] Setting key 1b504e4f48a086755a3976a9bcfb16e5
Sep 27 15:19:20 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:20.042708 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.022707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40567 (local address 127.2.246.126:42615)
0927 15:19:20.023340 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.023346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.024735 (+ 1389us) server_negotiation.cc:407] Connection header received
0927 15:19:20.026915 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.026920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.027025 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.027153 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:20.029147 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.030139 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.031738 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.032173 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.032313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.036252 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:20.036257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:20.036261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:20.036299 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:20.039342 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.040296 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.040322 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.040325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.040442 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.040897 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.040904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.040907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.041182 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:20.041344 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.042100 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.042325 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":360,"threads_started":1}
I20260927 15:19:20.043865 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.023600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42615 (local address 127.2.246.66:40567)
0927 15:19:20.024492 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.024542 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.026602 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.027463 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.027475 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.028188 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:20.028981 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.029001 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.030284 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.030289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.031574 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.031585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.032413 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.032420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.032649 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.033532 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:20.033563 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:20.035980 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:20.039514 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:20.039524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:20.039550 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:20.040077 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:20.040561 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:20.040566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:20.040569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:20.040778 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:20.041317 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:20.041325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:20.041692 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.042028 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.042588 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":345,"threads_started":1}
I20260927 15:19:20.045552 3932 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42615
I20260927 15:19:20.046162 3932 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:20.047390 3932 heartbeater.cc:511] Master 127.2.246.126:42615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:20.049561 3609 ts_manager.cc:194] Registered new tserver with Master: 04414fdb7a0d48debcd1b6d963ab190d (127.2.246.66:32885)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:19:20.050781 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:40567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:20.078770 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:20.392724 3798 heartbeater.cc:503] Master 127.2.246.126:42615 was elected leader, sending a full tablet report...
W20260927 15:19:20.462020 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:20.462409 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:20.462594 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:20.470002 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:20.470192 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:20.470388 3938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:20.480477 3938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:20.482004 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:20.483446 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:20.498348 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:20.498464 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:20.498416 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:20.499521 3938 server_base.cc:1034] running on GCE node
I20260927 15:19:20.500269 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:20.501364 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:20.502631 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790522360502573 us; error 53 us; skew 500 ppm
I20260927 15:19:20.502972 3938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:20 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:20.507009 3938 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:20.509696 3938 webserver.cc:460] Webserver started at http://127.2.246.67:39357/ using document root <none> and password file <none>
I20260927 15:19:20.510699 3938 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:20.510824 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:20.511175 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:20.515127 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "9d6c20d7509c4759bf3d0c4ffec016cc"
format_stamp: "Formatted at 2026-09-27 15:19:20 on dist-test-slave-14jq"
server_key: "1fd12efdd14d9fb751487cd53c392f10"
server_key_iv: "16229bcfe08cb0e378ff02149fd913fa"
server_key_version: "encryptionkey@0"
I20260927 15:19:20.516222 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "9d6c20d7509c4759bf3d0c4ffec016cc"
format_stamp: "Formatted at 2026-09-27 15:19:20 on dist-test-slave-14jq"
server_key: "1fd12efdd14d9fb751487cd53c392f10"
server_key_iv: "16229bcfe08cb0e378ff02149fd913fa"
server_key_version: "encryptionkey@0"
I20260927 15:19:20.523507 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 15:19:20.528823 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:20.531057 3938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:20.531343 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "9d6c20d7509c4759bf3d0c4ffec016cc"
format_stamp: "Formatted at 2026-09-27 15:19:20 on dist-test-slave-14jq"
server_key: "1fd12efdd14d9fb751487cd53c392f10"
server_key_iv: "16229bcfe08cb0e378ff02149fd913fa"
server_key_version: "encryptionkey@0"
I20260927 15:19:20.531612 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:20.552440 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:20.558779 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:20.559187 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:20.560604 3938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:20.562685 3938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:20.562806 3938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:20.563001 3938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:20.563084 3938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:20.610421 4065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:37027 every 8 connection(s)
I20260927 15:19:20.610441 3938 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:37027
I20260927 15:19:20.612718 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:20.614233 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 3938
I20260927 15:19:20.614461 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:20.615168 3033 external_mini_cluster.cc:1428] Setting key 35fb04d7fb67b59d7b6256ff1613053a
Sep 27 15:19:20 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:20.640813 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.616288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:49811 (local address 127.2.246.126:42615)
0927 15:19:20.616928 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.616937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.618942 (+ 2005us) server_negotiation.cc:407] Connection header received
0927 15:19:20.621819 (+ 2877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.621826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.621998 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.622159 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:20.624348 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.625194 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.627297 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.627737 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.627888 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.634165 (+ 6277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:20.634173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:20.634178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:20.634344 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:20.637339 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.638362 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.638368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.638372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.638512 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.638967 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.638972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.638977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.639267 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:20.639430 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.640187 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.640419 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":315,"threads_started":1}
I20260927 15:19:20.642004 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.617048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42615 (local address 127.2.246.67:49811)
0927 15:19:20.618724 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.618798 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.621401 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.622497 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.622510 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.623224 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:20.624133 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.624158 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.625380 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.625384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.627130 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.627146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.628013 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.628021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.628302 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.629323 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:20.629352 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:20.633795 (+ 4443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:20.637524 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:20.637534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:20.637555 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:20.638209 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:20.638630 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:20.638635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:20.638639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:20.638841 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:20.639399 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:20.639408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:20.639808 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.640091 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.640665 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":510,"threads_started":1}
I20260927 15:19:20.644085 4066 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42615
I20260927 15:19:20.644480 4066 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:20.645373 4066 heartbeater.cc:511] Master 127.2.246.126:42615 requested a full tablet report, sending...
I20260927 15:19:20.647257 3608 ts_manager.cc:194] Registered new tserver with Master: 9d6c20d7509c4759bf3d0c4ffec016cc (127.2.246.67:37027)
I20260927 15:19:20.648367 3608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:49811
I20260927 15:19:20.660777 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:20.683674 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.668800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.2.246.126:42615)
0927 15:19:20.669141 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.669151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.669571 (+ 420us) server_negotiation.cc:407] Connection header received
0927 15:19:20.669830 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.669847 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.669932 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.670052 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:20.672095 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.672884 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.674322 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.674763 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.674913 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.677575 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:20.677581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:20.677585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:20.677662 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:20.680681 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.681502 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.681509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.681513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.681648 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:20.682111 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:20.682118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:20.682121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:20.682473 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:20.682727 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.683079 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.683343 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 15:19:20.692282 3608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:20.700857 3608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:20.731364 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.717922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.2.246.66:32885)
0927 15:19:20.720775 (+ 2853us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.720792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.720846 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:19:20.721005 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.721020 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.721482 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.721810 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:20.723064 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.724780 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.728103 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.729710 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.729909 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.730121 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.730818 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.731008 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2629,"thread_start_us":2431,"threads_started":1}
I20260927 15:19:20.732084 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.717964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:32885 (local address 127.0.0.1:33826)
0927 15:19:20.719503 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.719575 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.719911 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.721887 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.721900 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.721948 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:20.722620 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.722639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.725758 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.725768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.727848 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.727866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.730036 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.730049 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.730324 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.730536 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.730735 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.731083 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":4142,"threads_started":1}
I20260927 15:19:20.733819 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.718836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.2.246.65:43161)
0927 15:19:20.723478 (+ 4642us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.723489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.723532 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:19:20.723645 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.723655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.723963 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.724206 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:20.726252 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.727954 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.730441 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.732175 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.732319 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.732537 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.733334 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.733529 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4418,"thread_start_us":4174,"threads_started":1}
I20260927 15:19:20.735749 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.718156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:43161 (local address 127.0.0.1:58708)
0927 15:19:20.721189 (+ 3033us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.721211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.721355 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.724536 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.724541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.724571 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:20.725943 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.725963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.728163 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.728170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.730213 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.730228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.732746 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.732753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.732972 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.733008 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.733134 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.733337 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2873,"mutex_wait_us":1,"thread_start_us":844,"threads_started":1}
I20260927 15:19:20.737263 3867 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7d0aeecc37467da0e564679d204105 (DEFAULT_TABLE table=test-table [id=c8db192b3ebc450a87cce2b2a86168c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:20.739028 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.720543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37027 (local address 127.0.0.1:35650)
0927 15:19:20.722546 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.722574 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.722719 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.727875 (+ 5156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.727881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.727915 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:20.728552 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.728567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.730637 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.730646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.734679 (+ 4033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.734698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.738008 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.738018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.738244 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.738296 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.738406 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.738594 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"mutex_wait_us":1000,"thread_start_us":2707,"threads_started":1}
I20260927 15:19:20.739216 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7d0aeecc37467da0e564679d204105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:20.740094 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.721250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.2.246.67:37027)
0927 15:19:20.726418 (+ 5168us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.726431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.726478 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:19:20.726608 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.726618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.726937 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.727204 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:20.728795 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.730372 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.734884 (+ 4512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.737517 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.737690 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.739489 (+ 1799us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.739581 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.739760 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4938,"thread_start_us":4734,"threads_started":1}
I20260927 15:19:20.741112 3733 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7d0aeecc37467da0e564679d204105 (DEFAULT_TABLE table=test-table [id=c8db192b3ebc450a87cce2b2a86168c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:20.743902 3733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7d0aeecc37467da0e564679d204105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:20.747179 4001 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7d0aeecc37467da0e564679d204105 (DEFAULT_TABLE table=test-table [id=c8db192b3ebc450a87cce2b2a86168c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:20.749915 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7d0aeecc37467da0e564679d204105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:20.753485 4085 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Bootstrap starting.
I20260927 15:19:20.759900 4085 tablet_bootstrap.cc:654] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:20.760927 4086 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Bootstrap starting.
I20260927 15:19:20.762301 4085 log.cc:824] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:20.765333 4086 tablet_bootstrap.cc:654] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:20.766389 4085 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: No bootstrap required, opened a new log
I20260927 15:19:20.766772 4085 ts_tablet_manager.cc:1397] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260927 15:19:20.766772 4086 log.cc:824] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:20.769459 4089 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Bootstrap starting.
I20260927 15:19:20.775545 4089 tablet_bootstrap.cc:654] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:20.777725 4089 log.cc:824] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:20.777868 4086 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: No bootstrap required, opened a new log
I20260927 15:19:20.778244 4086 ts_tablet_manager.cc:1397] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260927 15:19:20.779459 4085 raft_consensus.cc:348] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.779968 4085 raft_consensus.cc:374] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:20.780059 4085 raft_consensus.cc:729] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04414fdb7a0d48debcd1b6d963ab190d, State: Initialized, Role: FOLLOWER
I20260927 15:19:20.780838 4085 consensus_queue.cc:260] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.787278 3932 heartbeater.cc:503] Master 127.2.246.126:42615 was elected leader, sending a full tablet report...
I20260927 15:19:20.788156 4085 ts_tablet_manager.cc:1428] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260927 15:19:20.791059 4089 tablet_bootstrap.cc:492] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: No bootstrap required, opened a new log
I20260927 15:19:20.790784 4086 raft_consensus.cc:348] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.791227 4086 raft_consensus.cc:374] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:20.791322 4086 raft_consensus.cc:729] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09c8b41b41e429891f95c640133e21e, State: Initialized, Role: FOLLOWER
I20260927 15:19:20.791383 4089 ts_tablet_manager.cc:1397] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.003s
I20260927 15:19:20.792089 4086 consensus_queue.cc:260] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.794792 4086 ts_tablet_manager.cc:1428] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 15:19:20.800235 4089 raft_consensus.cc:348] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.800653 4089 raft_consensus.cc:374] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:20.800761 4089 raft_consensus.cc:729] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6c20d7509c4759bf3d0c4ffec016cc, State: Initialized, Role: FOLLOWER
I20260927 15:19:20.801319 4089 consensus_queue.cc:260] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.802630 4066 heartbeater.cc:503] Master 127.2.246.126:42615 was elected leader, sending a full tablet report...
I20260927 15:19:20.803085 4089 ts_tablet_manager.cc:1428] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
W20260927 15:19:20.850324 3799 tablet.cc:2369] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:20.865099 4067 tablet.cc:2369] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:20.969573 4093 raft_consensus.cc:482] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:20.970000 4093 raft_consensus.cc:504] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.972968 4093 leader_election.cc:288] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04414fdb7a0d48debcd1b6d963ab190d (127.2.246.66:32885), a09c8b41b41e429891f95c640133e21e (127.2.246.65:43161)
I20260927 15:19:20.980530 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.973447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:47449 (local address 127.2.246.66:32885)
0927 15:19:20.973947 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.973954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.974047 (+ 93us) server_negotiation.cc:407] Connection header received
0927 15:19:20.974322 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.974328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.974403 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.974534 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:20.975027 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.975902 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.978286 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.979373 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.979504 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.979559 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.980084 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.980256 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260927 15:19:20.980991 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.973441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:32885 (local address 127.2.246.67:47449)
0927 15:19:20.973946 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.973974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.974151 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.974618 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.974625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.974648 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:20.974868 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.974877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.976094 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.976102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.978103 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.978118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.979590 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.979598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.979848 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.979896 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.980030 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.980219 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333}
I20260927 15:19:20.981415 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.973536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:43161 (local address 127.2.246.67:40527)
0927 15:19:20.974393 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.974411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:20.974614 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:20.975019 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:20.975025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:20.975047 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:20.975508 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.975519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.976688 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.976694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.978691 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:20.978710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.980568 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.980576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:20.980776 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.980806 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:19:20.980892 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:19:20.981008 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":699,"threads_started":1}
I20260927 15:19:20.981419 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.973639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:40527 (local address 127.2.246.65:43161)
0927 15:19:20.973976 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:20.973987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:20.974530 (+ 543us) server_negotiation.cc:407] Connection header received
0927 15:19:20.974809 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:20.974814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:20.974886 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:20.975004 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:20.975690 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.976491 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.978960 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:20.980289 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:20.980487 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:20.980571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:20.980933 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 15:19:20.981114 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:19:20.982466 3887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7d0aeecc37467da0e564679d204105" candidate_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04414fdb7a0d48debcd1b6d963ab190d" is_pre_election: true
I20260927 15:19:20.982914 3753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7d0aeecc37467da0e564679d204105" candidate_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09c8b41b41e429891f95c640133e21e" is_pre_election: true
I20260927 15:19:20.983108 3887 raft_consensus.cc:2454] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6c20d7509c4759bf3d0c4ffec016cc in term 0.
I20260927 15:19:20.983543 3753 raft_consensus.cc:2454] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6c20d7509c4759bf3d0c4ffec016cc in term 0.
I20260927 15:19:20.984126 3956 leader_election.cc:302] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04414fdb7a0d48debcd1b6d963ab190d, 9d6c20d7509c4759bf3d0c4ffec016cc; no voters:
I20260927 15:19:20.984673 4093 raft_consensus.cc:2781] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:20.984841 4093 raft_consensus.cc:482] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:20.984970 4093 raft_consensus.cc:3037] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:20.987694 4093 raft_consensus.cc:504] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.988552 4093 leader_election.cc:288] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [CANDIDATE]: Term 1 election: Requested vote from peers 04414fdb7a0d48debcd1b6d963ab190d (127.2.246.66:32885), a09c8b41b41e429891f95c640133e21e (127.2.246.65:43161)
I20260927 15:19:20.989106 3887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7d0aeecc37467da0e564679d204105" candidate_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04414fdb7a0d48debcd1b6d963ab190d"
I20260927 15:19:20.989297 3753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7d0aeecc37467da0e564679d204105" candidate_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09c8b41b41e429891f95c640133e21e"
I20260927 15:19:20.989378 3887 raft_consensus.cc:3037] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:20.989580 3753 raft_consensus.cc:3037] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:20.992121 3887 raft_consensus.cc:2454] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6c20d7509c4759bf3d0c4ffec016cc in term 1.
I20260927 15:19:20.992761 3956 leader_election.cc:302] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04414fdb7a0d48debcd1b6d963ab190d, 9d6c20d7509c4759bf3d0c4ffec016cc; no voters:
I20260927 15:19:20.993175 4093 raft_consensus.cc:2781] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:20.993479 3753 raft_consensus.cc:2454] T 9f7d0aeecc37467da0e564679d204105 P a09c8b41b41e429891f95c640133e21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6c20d7509c4759bf3d0c4ffec016cc in term 1.
I20260927 15:19:20.993875 4093 raft_consensus.cc:686] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [term 1 LEADER]: Becoming Leader. State: Replica: 9d6c20d7509c4759bf3d0c4ffec016cc, State: Running, Role: LEADER
I20260927 15:19:20.994532 4093 consensus_queue.cc:237] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } }
I20260927 15:19:20.999650 3608 catalog_manager.cc:5495] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d6c20d7509c4759bf3d0c4ffec016cc (127.2.246.67). New cstate: current_term: 1 leader_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04414fdb7a0d48debcd1b6d963ab190d" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 32885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a09c8b41b41e429891f95c640133e21e" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 43161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d6c20d7509c4759bf3d0c4ffec016cc" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37027 } health_report { overall_health: HEALTHY } } }
W20260927 15:19:21.021646 3933 tablet.cc:2369] T 9f7d0aeecc37467da0e564679d204105 P 04414fdb7a0d48debcd1b6d963ab190d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:21.049988 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.030200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.2.246.65:43161)
0927 15:19:21.030477 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.030489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.031240 (+ 751us) server_negotiation.cc:407] Connection header received
0927 15:19:21.031567 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.031571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.031657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.031757 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.034424 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.035309 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.037185 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.037808 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.037993 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.043195 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.043214 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.043232 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.043283 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.046733 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.047603 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.047612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.047615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.047760 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.048191 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.048201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.048204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.048594 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.048924 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.049273 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.049516 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:21.106602 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.087556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.2.246.65:43161)
0927 15:19:21.087848 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.087855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.088267 (+ 412us) server_negotiation.cc:407] Connection header received
0927 15:19:21.088379 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.088383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.088462 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.088572 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.091314 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.092199 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.093765 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.094367 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.094493 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.099331 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.099338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.099342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.099395 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.102986 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.103857 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.103862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.103864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.104031 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.104562 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.104569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.104572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.104994 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.105293 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.105971 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.106204 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:21.165043 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.146903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.2.246.65:43161)
0927 15:19:21.147213 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.147219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.147362 (+ 143us) server_negotiation.cc:407] Connection header received
0927 15:19:21.147642 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.147649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.147742 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.147858 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.149428 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.150309 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.151694 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.152117 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.152245 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.158056 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.158063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.158068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.158130 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.161609 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.162698 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.162704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.162707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.162887 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.163392 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.163396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.163398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.163859 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.164096 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.164469 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.164751 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260927 15:19:21.167155 3713 tablet_service.cc:2908] Rejecting scan request for tablet 9f7d0aeecc37467da0e564679d204105: Uninitialized: clean time has not yet been initialized
I20260927 15:19:21.171178 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3670
I20260927 15:19:21.186363 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.172340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.2.246.65:43161)
0927 15:19:21.172553 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.172557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.172902 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:19:21.173064 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.173068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.173197 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.173357 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.174854 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.175545 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.176776 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.177179 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.177320 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.179964 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.179969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.179973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.180009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.183143 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.183926 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.183935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.183941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.184110 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.184823 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.184827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.184830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.185149 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.185425 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.185813 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.186089 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:19:21.187294 3793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:21.280467 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3670
I20260927 15:19:21.298560 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3804
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:21.324803 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.299398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.2.246.66:32885)
0927 15:19:21.299626 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.299633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.299655 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:21.299825 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.299830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.299907 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.300037 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.301400 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.302459 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.305263 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.306387 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.306551 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.313451 (+ 6900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.313503 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.313543 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.313620 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.319102 (+ 5482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.320168 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.320184 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.320191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.320435 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.321039 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.321046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.321051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.321749 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.322394 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.324012 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.324334 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:19:21.325884 3927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:21.438026 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3804
I20260927 15:19:21.459441 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3938
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:21.483294 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.482310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:43161 (local address 127.2.246.67:51175)
0927 15:19:21.482876 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.482995 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:43161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
W20260927 15:19:21.483819 3955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.65:43161: connect: Connection refused (error 111)
I20260927 15:19:21.485036 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.460450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.2.246.67:37027)
0927 15:19:21.460931 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:21.460936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:21.460956 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:19:21.461014 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:21.461018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:21.461084 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:21.461192 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:21.462806 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.463547 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.465204 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:21.465661 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:21.465801 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:21.477447 (+ 11646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:21.477465 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:21.477495 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:21.477539 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:21.480737 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.481759 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.481770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.481773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.482086 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:21.482684 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:21.482695 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:21.482699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:21.483258 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:21.483579 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:21.484183 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 15:19:21.484482 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
I20260927 15:19:21.485806 4061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:21.486331 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.485729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:43161 (local address 127.2.246.67:58123)
0927 15:19:21.485943 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.486049 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:43161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 15:19:21.486658 3955 consensus_peers.cc:487] T 9f7d0aeecc37467da0e564679d204105 P 9d6c20d7509c4759bf3d0c4ffec016cc -> Peer a09c8b41b41e429891f95c640133e21e (127.2.246.65:43161): Couldn't send request to peer a09c8b41b41e429891f95c640133e21e. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.65:43161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260927 15:19:21.588449 3955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.246.65:43161, user_credentials={real_user=slave}} blocked reactor thread for 100063us
I20260927 15:19:21.589108 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.588526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:32885 (local address 127.2.246.67:39473)
0927 15:19:21.588752 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.588866 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.66:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 15:19:21.589243 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3938
I20260927 15:19:21.606768 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 3577
I20260927 15:19:21.608520 3639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:21.736150 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 3577
2026-09-27T15:19:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5021 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[4117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[4117](info): set up 2 sockets
Sep 27 15:19:21 dist-test-slave-14jq krb5kdc[4117](info): commencing operation
krb5kdc: starting...
W20260927 15:19:22.862519 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:23 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:23.077458 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:36439
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:23.422736 4130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:23.423063 4130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:23.423202 4130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:23.430972 4130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:23.431102 4130 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:19:23.431140 4130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:23.431247 4130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:23.431325 4130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:23.431407 4130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:23.441751 4130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:36439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:23.443202 4130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:23.444643 4130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:23.457872 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:23.457877 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:23.457888 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:23.459658 4130 server_base.cc:1034] running on GCE node
I20260927 15:19:23.460605 4130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:23.462160 4130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:23.463418 4130 hybrid_clock.cc:648] HybridClock initialized: now 1790522363463337 us; error 57 us; skew 500 ppm
I20260927 15:19:23.463795 4130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:23 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522363, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:23.467396 4130 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:23.469151 4130 webserver.cc:460] Webserver started at http://127.2.246.126:45635/ using document root <none> and password file <none>
I20260927 15:19:23.469976 4130 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:23.470104 4130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:23.470451 4130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:23.473445 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "f2ed80c8895840b396bc93b0e2bfc4a2"
format_stamp: "Formatted at 2026-09-27 15:19:23 on dist-test-slave-14jq"
server_key: "ec119fc241e65c92839638ea4d092426"
server_key_iv: "9a385101b315d0a9660b4b4d9924da3c"
server_key_version: "encryptionkey@0"
I20260927 15:19:23.474323 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "f2ed80c8895840b396bc93b0e2bfc4a2"
format_stamp: "Formatted at 2026-09-27 15:19:23 on dist-test-slave-14jq"
server_key: "ec119fc241e65c92839638ea4d092426"
server_key_iv: "9a385101b315d0a9660b4b4d9924da3c"
server_key_version: "encryptionkey@0"
I20260927 15:19:23.479836 4130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 15:19:23.483784 4145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:23.485458 4130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:19:23.485688 4130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "f2ed80c8895840b396bc93b0e2bfc4a2"
format_stamp: "Formatted at 2026-09-27 15:19:23 on dist-test-slave-14jq"
server_key: "ec119fc241e65c92839638ea4d092426"
server_key_iv: "9a385101b315d0a9660b4b4d9924da3c"
server_key_version: "encryptionkey@0"
I20260927 15:19:23.485965 4130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:23.502310 4130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:23.507527 4130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:23.507846 4130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:23.532855 4130 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:36439
I20260927 15:19:23.532897 4206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:36439 every 8 connection(s)
I20260927 15:19:23.534376 4130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:23.539237 4207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:23.541711 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4130
I20260927 15:19:23.542027 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:23.542896 3033 external_mini_cluster.cc:1428] Setting key c63bb5e86bcc76b8a9bc12c067230e0c
I20260927 15:19:23.554687 4207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Bootstrap starting.
I20260927 15:19:23.559752 4207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Neither blocks nor log segments found. Creating new log.
Sep 27 15:19:23 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:23.561264 4207 log.cc:824] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:23.565814 4207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: No bootstrap required, opened a new log
I20260927 15:19:23.572607 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:23.545737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.2.246.126:36439)
0927 15:19:23.546846 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:23.546861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:23.546918 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:19:23.549087 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:23.549115 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:23.549638 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:23.550207 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:23.552204 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:23.553796 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:23.555777 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:23.556589 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:23.556778 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:23.562638 (+ 5860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:23.562650 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:23.562673 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:23.562728 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:23.567083 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:23.567906 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:23.567916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:23.567926 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:23.568199 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:23.568589 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:23.568593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:23.568597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:23.569454 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:23.569800 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:23.570329 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:19:23.570898 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":476,"threads_started":1}
I20260927 15:19:23.577021 4207 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } }
I20260927 15:19:23.577472 4207 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:23.577557 4207 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ed80c8895840b396bc93b0e2bfc4a2, State: Initialized, Role: FOLLOWER
I20260927 15:19:23.578305 4207 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } }
I20260927 15:19:23.578552 4207 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:23.578665 4207 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:23.578845 4207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:23.581664 4207 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } }
I20260927 15:19:23.582319 4207 leader_election.cc:302] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2ed80c8895840b396bc93b0e2bfc4a2; no voters:
I20260927 15:19:23.582947 4207 leader_election.cc:288] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:23.583177 4212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:23.584060 4212 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [term 1 LEADER]: Becoming Leader. State: Replica: f2ed80c8895840b396bc93b0e2bfc4a2, State: Running, Role: LEADER
I20260927 15:19:23.584862 4212 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } }
I20260927 15:19:23.585597 4207 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:23.588536 4213 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } } }
I20260927 15:19:23.588541 4214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2ed80c8895840b396bc93b0e2bfc4a2. Latest consensus state: current_term: 1 leader_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed80c8895840b396bc93b0e2bfc4a2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 36439 } } }
I20260927 15:19:23.589258 4214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:23.589258 4213 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2 [sys.catalog]: This master's current role is: LEADER
W20260927 15:19:23.598789 4227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:19:23.598932 4227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:19:23.599148 4221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:23.600863 4221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:23.610347 4221 catalog_manager.cc:1347] Generated new cluster ID: 15ae6af8364642b9aa927b745f6bc1d9
I20260927 15:19:23.610493 4221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:23.626933 4221 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:23.628397 4221 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:23.637025 4221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Generated new TSK 0
I20260927 15:19:23.638057 4221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:23.691545 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:36439
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:24.132212 4233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:24.132599 4233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:24.132766 4233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:24.140832 4233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:24.140987 4233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:24.141183 4233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:24.141285 4233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:24.153481 4233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:24.155423 4233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:24.157191 4233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:24.175309 4239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:24.175330 4238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:24.175427 4241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:24.176713 4233 server_base.cc:1034] running on GCE node
I20260927 15:19:24.177662 4233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:24.179008 4233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:24.180485 4233 hybrid_clock.cc:648] HybridClock initialized: now 1790522364180248 us; error 57 us; skew 500 ppm
I20260927 15:19:24.181021 4233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:24 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:24.187230 4233 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:24.189424 4233 webserver.cc:460] Webserver started at http://127.2.246.65:43105/ using document root <none> and password file <none>
I20260927 15:19:24.190601 4233 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:24.190775 4233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:24.191255 4233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:24.195444 4233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "d2725753eee04eb1884ccbb659979cc1"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "682359cf3825cc3f2fa77095db0af0a9"
server_key_iv: "0ec473c5d0dd31ae1180fa9052ec722f"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.196770 4233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "d2725753eee04eb1884ccbb659979cc1"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "682359cf3825cc3f2fa77095db0af0a9"
server_key_iv: "0ec473c5d0dd31ae1180fa9052ec722f"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.206120 4233 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20260927 15:19:24.212260 4248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:24.214601 4233 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:19:24.214926 4233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "d2725753eee04eb1884ccbb659979cc1"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "682359cf3825cc3f2fa77095db0af0a9"
server_key_iv: "0ec473c5d0dd31ae1180fa9052ec722f"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.215271 4233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:24.247351 4233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:24.253198 4233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:24.253731 4233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:24.257961 4233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:24.258133 4233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:19:24.258332 4233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:24.258419 4233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:24 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:24.293216 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.258297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:55185 (local address 127.2.246.126:36439)
0927 15:19:24.259009 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:24.259017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:24.260708 (+ 1691us) server_negotiation.cc:407] Connection header received
0927 15:19:24.264516 (+ 3808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:24.264523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:24.264629 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:24.264826 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:24.267877 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.268925 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.272091 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.272834 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.273096 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.281666 (+ 8570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:24.281678 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:24.281686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:24.281758 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:24.286814 (+ 5056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.288603 (+ 1789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.288610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.288613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.288811 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.289736 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.289746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.289752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.290639 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:24.290828 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:24.292589 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0927 15:19:24.292896 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":336,"threads_started":1}
I20260927 15:19:24.295194 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.259213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.65:55185)
0927 15:19:24.260413 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.260484 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:24.263927 (+ 3443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:24.265371 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:24.265391 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:24.266400 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:24.267590 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.267636 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.269194 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.269198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.271866 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.271885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.273266 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.273277 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.273646 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.274942 (+ 1296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:24.274990 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:24.281098 (+ 6108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:24.287106 (+ 6008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.287123 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.287158 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.288273 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.288985 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.288994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.289000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.289393 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.290823 (+ 1430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:24.290882 (+ 59us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:24.291716 (+ 834us) client_negotiation.cc:769] Sending connection context
0927 15:19:24.292169 (+ 453us) client_negotiation.cc:240] Negotiation successful
0927 15:19:24.293027 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":413,"threads_started":1}
I20260927 15:19:24.330137 4233 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:38183
I20260927 15:19:24.330175 4368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:38183 every 8 connection(s)
I20260927 15:19:24.330377 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.305691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.65:37633)
0927 15:19:24.310465 (+ 4774us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.310501 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:24.310784 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:24.311220 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:24.311225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:24.311844 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:24.312208 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.312219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.313352 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.313360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.315025 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.315052 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.316489 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.316494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.316719 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.318107 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:24.318150 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:24.319644 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:24.325719 (+ 6075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.325726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.325730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.326636 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.327440 (+ 804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.327447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.327451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.327680 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.328422 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:24.328428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:24.328705 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 15:19:24.329168 (+ 463us) client_negotiation.cc:240] Negotiation successful
0927 15:19:24.329660 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4565,"thread_start_us":4394,"threads_started":1}
I20260927 15:19:24.330932 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.305690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:37633 (local address 127.2.246.126:36439)
0927 15:19:24.305976 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:24.305985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:24.310719 (+ 4734us) server_negotiation.cc:407] Connection header received
0927 15:19:24.310958 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:24.310966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:24.311063 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:24.311188 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:24.312390 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.313193 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.315221 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.315932 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.316137 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.320385 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:24.320398 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:24.320408 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:24.320471 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:24.325355 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.326861 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.326867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.326869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.327228 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.327884 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.327888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.327890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.328252 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:24.328488 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:24.330205 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0927 15:19:24.330545 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:19:24.332103 4233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:24.335505 4369 heartbeater.cc:347] Connected to a master server at 127.2.246.126:36439
I20260927 15:19:24.335840 4369 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:24.336896 4369 heartbeater.cc:511] Master 127.2.246.126:36439 requested a full tablet report, sending...
I20260927 15:19:24.339783 4162 ts_manager.cc:194] Registered new tserver with Master: d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183)
I20260927 15:19:24.342033 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4233
I20260927 15:19:24.342317 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:24.342475 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:55185
I20260927 15:19:24.343343 3033 external_mini_cluster.cc:1428] Setting key 420973e5120fe615058d5abff120da83
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:24.393625 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:36439
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:24.765645 4373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:24.766028 4373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:24.766211 4373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:24.773777 4373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:24.773960 4373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:24.774101 4373 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:24.774252 4373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:24.784695 4373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:24.786258 4373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:24.787684 4373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:24.802701 4381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:24.802685 4378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:24.803500 4379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:24.807901 4373 server_base.cc:1034] running on GCE node
I20260927 15:19:24.808864 4373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:24.810606 4373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:24.811960 4373 hybrid_clock.cc:648] HybridClock initialized: now 1790522364811835 us; error 79 us; skew 500 ppm
I20260927 15:19:24.812435 4373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:24 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:24.817114 4373 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:24.819494 4373 webserver.cc:460] Webserver started at http://127.2.246.66:44619/ using document root <none> and password file <none>
I20260927 15:19:24.820586 4373 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:24.820716 4373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:24.821051 4373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:24.824903 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "1f87eca6cc464749872737ae59599c34"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "ab6045bef63efebc42c420152181e3e1"
server_key_iv: "f4ddbe22f7ac42790430513870fe216b"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.825977 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "1f87eca6cc464749872737ae59599c34"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "ab6045bef63efebc42c420152181e3e1"
server_key_iv: "f4ddbe22f7ac42790430513870fe216b"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.833658 4373 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 15:19:24.839211 4388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:24.841471 4373 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260927 15:19:24.841825 4373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "1f87eca6cc464749872737ae59599c34"
format_stamp: "Formatted at 2026-09-27 15:19:24 on dist-test-slave-14jq"
server_key: "ab6045bef63efebc42c420152181e3e1"
server_key_iv: "f4ddbe22f7ac42790430513870fe216b"
server_key_version: "encryptionkey@0"
I20260927 15:19:24.842239 4373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:24.874507 4373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:24.880069 4373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:24.880427 4373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:24.884136 4373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:24.884297 4373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:24.884541 4373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:24.884618 4373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:24 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:24.914642 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.884643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:52603 (local address 127.2.246.126:36439)
0927 15:19:24.885193 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:24.885201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:24.886749 (+ 1548us) server_negotiation.cc:407] Connection header received
0927 15:19:24.890376 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:24.890385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:24.890524 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:24.890666 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:24.894599 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.897014 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.899145 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.899786 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.899927 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.906709 (+ 6782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:24.906720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:24.906727 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:24.906781 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:24.910707 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.911752 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.911759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.911763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.911931 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.912585 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.912593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.912597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.912935 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:24.913117 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:24.913988 (+ 871us) server_negotiation.cc:299] Negotiation successful
0927 15:19:24.914235 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260927 15:19:24.916136 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.885299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.66:52603)
0927 15:19:24.886500 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.886570 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:24.889748 (+ 3178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:24.891224 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:24.891248 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:24.892575 (+ 1327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:24.894247 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.894292 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.897228 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.897234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.898946 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.898968 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.900353 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.900363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.900788 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.902006 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:24.902034 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:24.906372 (+ 4338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:24.910945 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.910958 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.910993 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.911557 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.912086 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.912094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.912099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.912394 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.913093 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:24.913102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:24.913539 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 15:19:24.913840 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 15:19:24.914510 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":493,"threads_started":1}
I20260927 15:19:24.940038 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.924005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:37573 (local address 127.2.246.126:36439)
0927 15:19:24.924320 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:24.924330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:24.924691 (+ 361us) server_negotiation.cc:407] Connection header received
0927 15:19:24.924967 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:24.924976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:24.925096 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:24.925288 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:24.927104 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.928680 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.930373 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.930836 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.930990 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.933235 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:24.933240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:24.933244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:24.933273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:24.936126 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.937221 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.937228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.937234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.937380 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:24.937796 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:24.937802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:24.937805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:24.938172 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:24.938382 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:24.939324 (+ 942us) server_negotiation.cc:299] Negotiation successful
0927 15:19:24.939613 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 15:19:24.940659 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.923843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.66:37573)
0927 15:19:24.924545 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.924569 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:24.924769 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:24.925347 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:24.925358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:24.926246 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:24.926854 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.926876 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.928913 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.928917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.930165 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:24.930183 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:24.931060 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:24.931066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:24.931238 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:24.932099 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:24.932116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:24.933009 (+ 893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:24.936268 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.936273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.936278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.937015 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.937536 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:24.937540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:24.937544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:24.937684 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:24.938394 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:24.938406 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:24.938847 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 15:19:24.939178 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 15:19:24.939814 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":351,"threads_started":1}
I20260927 15:19:24.949077 4373 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:41549
I20260927 15:19:24.949113 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:41549 every 8 connection(s)
I20260927 15:19:24.950750 4373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:24.953527 4509 heartbeater.cc:347] Connected to a master server at 127.2.246.126:36439
I20260927 15:19:24.953938 4509 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:24.954814 4509 heartbeater.cc:511] Master 127.2.246.126:36439 requested a full tablet report, sending...
I20260927 15:19:24.956555 4162 ts_manager.cc:194] Registered new tserver with Master: 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549)
I20260927 15:19:24.957484 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:52603
I20260927 15:19:24.958304 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4373
I20260927 15:19:24.958546 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:24.959059 3033 external_mini_cluster.cc:1428] Setting key 814a6f94dc14d49668ee0a3f0babc9cb
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:25.001926 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:36439
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:25.347402 4369 heartbeater.cc:503] Master 127.2.246.126:36439 was elected leader, sending a full tablet report...
W20260927 15:19:25.349327 4513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:25.349778 4513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:25.349998 4513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:25.358444 4513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:25.358628 4513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:25.358791 4513 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:25.358968 4513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:25.370302 4513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:36439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:25.371770 4513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:25.373106 4513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:25.387511 4519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:25.388033 4518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:25.388230 4521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:25.394479 4513 server_base.cc:1034] running on GCE node
I20260927 15:19:25.395329 4513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:25.396545 4513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:25.398002 4513 hybrid_clock.cc:648] HybridClock initialized: now 1790522365397897 us; error 69 us; skew 500 ppm
I20260927 15:19:25.398442 4513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:25 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:25.402269 4513 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:25.403977 4513 webserver.cc:460] Webserver started at http://127.2.246.67:42203/ using document root <none> and password file <none>
I20260927 15:19:25.404691 4513 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:25.404779 4513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:25.405041 4513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:25.408063 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "eafe060205014c3d9e11cffdfe583f84"
format_stamp: "Formatted at 2026-09-27 15:19:25 on dist-test-slave-14jq"
server_key: "da4582e54208862d3418e2d4ee9bd04c"
server_key_iv: "f73f4e5ec7cfbb52a0dfd9e5e5208d69"
server_key_version: "encryptionkey@0"
I20260927 15:19:25.408905 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "eafe060205014c3d9e11cffdfe583f84"
format_stamp: "Formatted at 2026-09-27 15:19:25 on dist-test-slave-14jq"
server_key: "da4582e54208862d3418e2d4ee9bd04c"
server_key_iv: "f73f4e5ec7cfbb52a0dfd9e5e5208d69"
server_key_version: "encryptionkey@0"
I20260927 15:19:25.415479 4513 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 15:19:25.420400 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:25.422524 4513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:25.422758 4513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "eafe060205014c3d9e11cffdfe583f84"
format_stamp: "Formatted at 2026-09-27 15:19:25 on dist-test-slave-14jq"
server_key: "da4582e54208862d3418e2d4ee9bd04c"
server_key_iv: "f73f4e5ec7cfbb52a0dfd9e5e5208d69"
server_key_version: "encryptionkey@0"
I20260927 15:19:25.423017 4513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:25.454805 4513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:25.459865 4513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:25.460161 4513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:25.463348 4513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:25.463469 4513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:25.463642 4513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:25.463721 4513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:25 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:25.496168 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.464876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:47081 (local address 127.2.246.126:36439)
0927 15:19:25.465499 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.465506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.467170 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 15:19:25.470498 (+ 3328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.470508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.470654 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.470850 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:25.474416 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.475703 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.479296 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.480048 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.480253 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.486848 (+ 6595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:25.486859 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:25.486865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:25.486914 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:25.491588 (+ 4674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.493115 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.493121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.493125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.493296 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.493861 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.493868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.493875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.494344 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:25.494523 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.495435 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.495712 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":293,"threads_started":1}
I20260927 15:19:25.497550 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.465669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.67:47081)
0927 15:19:25.466916 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.466988 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.469955 (+ 2967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.471239 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.471259 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.472701 (+ 1442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:25.474088 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.474133 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.475970 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.475979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.478947 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.478982 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.480648 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.480658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.481165 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.482346 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:25.482375 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:25.486441 (+ 4066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:25.491905 (+ 5464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:25.491922 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:25.491954 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:25.492920 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:25.493427 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:25.493437 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:25.493441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:25.493698 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:25.494517 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:25.494533 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:25.495022 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.495339 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.496031 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":509,"threads_started":1}
I20260927 15:19:25.526335 4648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:46167 every 8 connection(s)
I20260927 15:19:25.526381 4513 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:46167
I20260927 15:19:25.527372 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.506979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:33469 (local address 127.2.246.126:36439)
0927 15:19:25.507263 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.507268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.507993 (+ 725us) server_negotiation.cc:407] Connection header received
0927 15:19:25.508209 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.508218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.508347 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.508496 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:25.509939 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.511237 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.514133 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.514862 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.515053 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.518665 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:25.518674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:25.518682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:25.518742 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:25.523194 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.524461 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.524465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.524468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.524780 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.525395 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.525401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.525406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.525870 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:25.526052 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.526785 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.527028 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 15:19:25.527694 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.506912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.2.246.67:33469)
0927 15:19:25.507794 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.507826 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.508030 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.508591 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.508595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.509249 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:25.509651 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.509664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.511578 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.511592 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.513820 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.513896 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.515179 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.515188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.515407 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.516307 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:25.516343 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:25.518314 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:25.523472 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:25.523479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:25.523484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:25.524265 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:25.524954 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:25.524962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:25.524969 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:25.525217 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:25.526039 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:25.526049 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:25.526393 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.526665 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.527097 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":520,"threads_started":1}
I20260927 15:19:25.528667 4513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:25.531927 4649 heartbeater.cc:347] Connected to a master server at 127.2.246.126:36439
I20260927 15:19:25.532312 4649 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:25.533365 4649 heartbeater.cc:511] Master 127.2.246.126:36439 requested a full tablet report, sending...
I20260927 15:19:25.534888 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4513
I20260927 15:19:25.535127 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:25.535444 4162 ts_manager.cc:194] Registered new tserver with Master: eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167)
I20260927 15:19:25.535627 3033 external_mini_cluster.cc:1428] Setting key f06fa8cf6822ac071e32c8fec4b1fa66
I20260927 15:19:25.536504 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:47081
I20260927 15:19:25.539773 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:19:25 dist-test-slave-14jq krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:19:25 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:25.597219 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.578367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.2.246.126:36439)
0927 15:19:25.578580 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.578585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.578985 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:19:25.579275 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.579281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.579367 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.579497 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:25.581151 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.582215 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.584073 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.584896 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.585070 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.590384 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:25.590396 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:25.590401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:25.590477 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:25.593666 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.594758 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.594763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.594766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.594970 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:25.595514 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:25.595518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:25.595520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:25.595903 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:25.596134 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.596648 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.596872 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:19:25.603669 4161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:25.612560 4161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:25.644410 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.629746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:38183 (local address 127.0.0.1:40018)
0927 15:19:25.632151 (+ 2405us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.632220 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.632464 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.635524 (+ 3060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.635529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.635550 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:25.636532 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.636551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.638636 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.638641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.641136 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.641158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.643398 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.643405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.643584 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.643613 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.643757 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.643983 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2104,"mutex_wait_us":9,"thread_start_us":3461,"threads_started":1}
I20260927 15:19:25.644409 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.631661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.2.246.66:41549)
0927 15:19:25.633960 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.633971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.634004 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:19:25.634114 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.634125 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.634480 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.635778 (+ 1298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:25.636167 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.637698 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.641118 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.642650 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.642808 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.643048 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.643920 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.644089 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065,"thread_start_us":1891,"threads_started":1}
I20260927 15:19:25.645128 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.630604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41549 (local address 127.0.0.1:39736)
0927 15:19:25.632949 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.632969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.633090 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.635259 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.635268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.635305 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:25.635891 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.635908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.637899 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.637905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.640918 (+ 3013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.640934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.643234 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.643241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.643515 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.643614 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.643757 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.644770 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"mutex_wait_us":939,"thread_start_us":5596,"threads_started":1}
I20260927 15:19:25.645231 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.630173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.2.246.65:38183)
0927 15:19:25.631451 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.631466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.632724 (+ 1258us) server_negotiation.cc:407] Connection header received
0927 15:19:25.632856 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.632867 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.634954 (+ 2087us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.635214 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:25.636797 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.638410 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.641405 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.642895 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.643036 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.643242 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.644740 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.644916 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":902,"threads_started":1}
I20260927 15:19:25.647583 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.630692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:46167 (local address 127.0.0.1:54034)
0927 15:19:25.636560 (+ 5868us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.636588 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.636723 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.637817 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.637825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.637898 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:25.638465 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.638483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.640703 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.640710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.643223 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.643239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.646039 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.646047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.646233 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.646267 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.646957 (+ 690us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.647137 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5679,"mutex_wait_us":4,"thread_start_us":998,"threads_started":1}
I20260927 15:19:25.647764 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.631102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.2.246.67:46167)
0927 15:19:25.635828 (+ 4726us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.635842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.636713 (+ 871us) server_negotiation.cc:407] Connection header received
0927 15:19:25.636883 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.636892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.637219 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.637483 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:25.638722 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.640478 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.643514 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.645390 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.645548 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.645842 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.647091 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.647325 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4435,"thread_start_us":3987,"threads_started":1}
I20260927 15:19:25.650314 4439 tablet_service.cc:1467] Processing CreateTablet for tablet 9673d79ba3e24753bc71f1f6d241745b (DEFAULT_TABLE table=test-table [id=b8251400bb1447bc94a45121fb785ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:25.650593 4299 tablet_service.cc:1467] Processing CreateTablet for tablet 9673d79ba3e24753bc71f1f6d241745b (DEFAULT_TABLE table=test-table [id=b8251400bb1447bc94a45121fb785ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:25.651674 4439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9673d79ba3e24753bc71f1f6d241745b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:25.652424 4299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9673d79ba3e24753bc71f1f6d241745b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:25.653134 4579 tablet_service.cc:1467] Processing CreateTablet for tablet 9673d79ba3e24753bc71f1f6d241745b (DEFAULT_TABLE table=test-table [id=b8251400bb1447bc94a45121fb785ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:25.654914 4579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9673d79ba3e24753bc71f1f6d241745b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:25.664885 4668 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Bootstrap starting.
I20260927 15:19:25.668264 4668 tablet_bootstrap.cc:654] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:25.669788 4668 log.cc:824] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:25.672863 4670 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Bootstrap starting.
I20260927 15:19:25.673215 4668 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: No bootstrap required, opened a new log
I20260927 15:19:25.673321 4671 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Bootstrap starting.
I20260927 15:19:25.673482 4668 ts_tablet_manager.cc:1397] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 15:19:25.677050 4670 tablet_bootstrap.cc:654] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:25.677063 4671 tablet_bootstrap.cc:654] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:25.678428 4670 log.cc:824] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:25.679451 4671 log.cc:824] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:25.682118 4670 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: No bootstrap required, opened a new log
I20260927 15:19:25.682495 4670 ts_tablet_manager.cc:1397] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 15:19:25.682832 4671 tablet_bootstrap.cc:492] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: No bootstrap required, opened a new log
I20260927 15:19:25.683197 4671 ts_tablet_manager.cc:1397] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 15:19:25.684376 4668 raft_consensus.cc:348] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.684855 4668 raft_consensus.cc:374] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:25.684955 4668 raft_consensus.cc:729] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f87eca6cc464749872737ae59599c34, State: Initialized, Role: FOLLOWER
I20260927 15:19:25.686075 4668 consensus_queue.cc:260] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.687470 4509 heartbeater.cc:503] Master 127.2.246.126:36439 was elected leader, sending a full tablet report...
I20260927 15:19:25.688068 4668 ts_tablet_manager.cc:1428] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260927 15:19:25.694032 4670 raft_consensus.cc:348] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.694522 4670 raft_consensus.cc:374] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:25.694619 4670 raft_consensus.cc:729] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2725753eee04eb1884ccbb659979cc1, State: Initialized, Role: FOLLOWER
I20260927 15:19:25.695338 4670 consensus_queue.cc:260] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.696251 4671 raft_consensus.cc:348] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.696820 4671 raft_consensus.cc:374] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:25.696944 4671 raft_consensus.cc:729] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafe060205014c3d9e11cffdfe583f84, State: Initialized, Role: FOLLOWER
I20260927 15:19:25.697608 4671 consensus_queue.cc:260] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.698540 4670 ts_tablet_manager.cc:1428] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 15:19:25.698881 4649 heartbeater.cc:503] Master 127.2.246.126:36439 was elected leader, sending a full tablet report...
I20260927 15:19:25.699481 4671 ts_tablet_manager.cc:1428] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
W20260927 15:19:25.703226 4510 tablet.cc:2369] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:25.781200 4650 tablet.cc:2369] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:25.835711 4370 tablet.cc:2369] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:25.874558 4674 raft_consensus.cc:482] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:25.875082 4674 raft_consensus.cc:504] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.878252 4674 leader_election.cc:288] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183), eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167)
I20260927 15:19:25.885919 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.878842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:34431 (local address 127.2.246.67:46167)
0927 15:19:25.879165 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.879177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.880088 (+ 911us) server_negotiation.cc:407] Connection header received
0927 15:19:25.880394 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.880399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.880475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.880578 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:25.881230 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.882085 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.884108 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.884890 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.885080 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.885161 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.885448 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.885588 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 15:19:25.886160 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.878712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:46167 (local address 127.2.246.66:34431)
0927 15:19:25.879955 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.879976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.880243 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.880596 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.880600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.880627 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:25.881077 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.881091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.882218 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.882226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.883949 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.883966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.885114 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.885120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.885291 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.885315 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.885425 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.885564 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":32,"thread_start_us":981,"threads_started":1}
I20260927 15:19:25.886693 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.878783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:33421 (local address 127.2.246.65:38183)
0927 15:19:25.879006 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.879015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.880273 (+ 1258us) server_negotiation.cc:407] Connection header received
0927 15:19:25.880538 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.880545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.880635 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.880754 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:25.881543 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.882416 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.884500 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.885363 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.885600 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.885698 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.886250 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.886450 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:19:25.887202 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9673d79ba3e24753bc71f1f6d241745b" candidate_uuid: "1f87eca6cc464749872737ae59599c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafe060205014c3d9e11cffdfe583f84" is_pre_election: true
I20260927 15:19:25.887511 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.878787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:38183 (local address 127.2.246.66:33421)
0927 15:19:25.880137 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.880156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:25.880316 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:25.880776 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:25.880781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:25.880811 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:25.881365 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.881376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.882585 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.882593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.884312 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:25.884326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.885630 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.885637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:25.885984 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.886050 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 15:19:25.886969 (+ 919us) client_negotiation.cc:240] Negotiation successful
0927 15:19:25.887137 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":29,"thread_start_us":486,"threads_started":1}
I20260927 15:19:25.887768 4599 raft_consensus.cc:2454] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f87eca6cc464749872737ae59599c34 in term 0.
I20260927 15:19:25.888554 4389 leader_election.cc:302] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f87eca6cc464749872737ae59599c34, eafe060205014c3d9e11cffdfe583f84; no voters:
I20260927 15:19:25.888669 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9673d79ba3e24753bc71f1f6d241745b" candidate_uuid: "1f87eca6cc464749872737ae59599c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2725753eee04eb1884ccbb659979cc1" is_pre_election: true
I20260927 15:19:25.889103 4674 raft_consensus.cc:2781] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:25.889163 4323 raft_consensus.cc:2454] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f87eca6cc464749872737ae59599c34 in term 0.
I20260927 15:19:25.889307 4674 raft_consensus.cc:482] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:25.889456 4674 raft_consensus.cc:3037] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:25.892709 4674 raft_consensus.cc:504] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.894392 4674 leader_election.cc:288] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [CANDIDATE]: Term 1 election: Requested vote from peers d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183), eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167)
I20260927 15:19:25.894380 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9673d79ba3e24753bc71f1f6d241745b" candidate_uuid: "1f87eca6cc464749872737ae59599c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2725753eee04eb1884ccbb659979cc1"
I20260927 15:19:25.894634 4323 raft_consensus.cc:3037] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:25.897045 4323 raft_consensus.cc:2454] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f87eca6cc464749872737ae59599c34 in term 1.
I20260927 15:19:25.897476 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9673d79ba3e24753bc71f1f6d241745b" candidate_uuid: "1f87eca6cc464749872737ae59599c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafe060205014c3d9e11cffdfe583f84"
I20260927 15:19:25.897624 4391 leader_election.cc:302] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f87eca6cc464749872737ae59599c34, d2725753eee04eb1884ccbb659979cc1; no voters:
I20260927 15:19:25.897706 4599 raft_consensus.cc:3037] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:25.898087 4674 raft_consensus.cc:2781] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:25.898665 4674 raft_consensus.cc:686] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 1 LEADER]: Becoming Leader. State: Replica: 1f87eca6cc464749872737ae59599c34, State: Running, Role: LEADER
I20260927 15:19:25.899734 4674 consensus_queue.cc:237] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } }
I20260927 15:19:25.900370 4599 raft_consensus.cc:2454] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f87eca6cc464749872737ae59599c34 in term 1.
I20260927 15:19:25.905484 4161 catalog_manager.cc:5495] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f87eca6cc464749872737ae59599c34 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "1f87eca6cc464749872737ae59599c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } health_report { overall_health: HEALTHY } } }
I20260927 15:19:25.954834 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.947774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.2.246.66:41549)
0927 15:19:25.947994 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:25.948001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:25.948206 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:19:25.948684 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:25.948689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:25.948776 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:25.948930 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:25.949321 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.950281 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.951748 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:25.952722 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:25.952861 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:25.953456 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:25.954098 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:25.954326 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:25.954377 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:19:25.954462 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:19:25.966302 4323 raft_consensus.cc:1261] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Refusing update from remote peer 1f87eca6cc464749872737ae59599c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:25.966447 4599 raft_consensus.cc:1261] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Refusing update from remote peer 1f87eca6cc464749872737ae59599c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:25.967592 4679 consensus_queue.cc:1035] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:25.968015 4674 consensus_queue.cc:1035] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:25.980510 4682 mvcc.cc:205] Tried to move back new op lower bound from 7333979610985754624 to 7333979610730815488. Current Snapshot: MvccSnapshot[applied={T|T < 7333979610985754624}]
I20260927 15:19:25.983829 4684 mvcc.cc:205] Tried to move back new op lower bound from 7333979610985754624 to 7333979610730815488. Current Snapshot: MvccSnapshot[applied={T|T < 7333979610985754624}]
I20260927 15:19:26.014952 4162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:26.015270 4162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:26.020830 4162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Sending DeleteTablet for 3 replicas of tablet 9673d79ba3e24753bc71f1f6d241745b
I20260927 15:19:26.022500 4579 tablet_service.cc:1514] Processing DeleteTablet for tablet 9673d79ba3e24753bc71f1f6d241745b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260927 15:19:26.022521 4439 tablet_service.cc:1514] Processing DeleteTablet for tablet 9673d79ba3e24753bc71f1f6d241745b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:39736
I20260927 15:19:26.022524 4299 tablet_service.cc:1514] Processing DeleteTablet for tablet 9673d79ba3e24753bc71f1f6d241745b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:40018
I20260927 15:19:26.026850 4162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:26.029033 4162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:26.044840 4578 tablet_service.cc:1467] Processing CreateTablet for tablet ae0556b3f9794da780de073b6bd3f59a (DEFAULT_TABLE table=test-table [id=595e1a7d34774b868830fe742cc1fbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:26.044938 4298 tablet_service.cc:1467] Processing CreateTablet for tablet ae0556b3f9794da780de073b6bd3f59a (DEFAULT_TABLE table=test-table [id=595e1a7d34774b868830fe742cc1fbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:26.045380 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0556b3f9794da780de073b6bd3f59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.045425 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0556b3f9794da780de073b6bd3f59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.051515 4438 tablet_service.cc:1467] Processing CreateTablet for tablet ae0556b3f9794da780de073b6bd3f59a (DEFAULT_TABLE table=test-table [id=595e1a7d34774b868830fe742cc1fbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:26.052110 4438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0556b3f9794da780de073b6bd3f59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.055087 4694 tablet_replica.cc:330] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: stopping tablet replica
I20260927 15:19:26.058163 4693 tablet_replica.cc:330] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: stopping tablet replica
I20260927 15:19:26.059082 4695 tablet_replica.cc:330] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: stopping tablet replica
I20260927 15:19:26.059155 4694 raft_consensus.cc:2229] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:26.059815 4694 raft_consensus.cc:2258] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.058202 4671 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Bootstrap starting.
I20260927 15:19:26.060513 4670 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Bootstrap starting.
I20260927 15:19:26.059789 4695 raft_consensus.cc:2229] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:26.061250 4695 raft_consensus.cc:2258] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.060405 4693 raft_consensus.cc:2229] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:26.062440 4693 raft_consensus.cc:2258] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.061744 4671 tablet_bootstrap.cc:654] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.063205 4668 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Bootstrap starting.
I20260927 15:19:26.064011 4670 tablet_bootstrap.cc:654] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.065182 4668 tablet_bootstrap.cc:654] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.072227 4695 ts_tablet_manager.cc:1905] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.074218 4693 ts_tablet_manager.cc:1905] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.075054 4694 ts_tablet_manager.cc:1905] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.085544 4668 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: No bootstrap required, opened a new log
I20260927 15:19:26.085824 4668 ts_tablet_manager.cc:1397] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.008s
I20260927 15:19:26.087638 4668 raft_consensus.cc:348] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.088722 4670 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: No bootstrap required, opened a new log
I20260927 15:19:26.088935 4670 ts_tablet_manager.cc:1397] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20260927 15:19:26.090152 4670 raft_consensus.cc:348] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.090433 4670 raft_consensus.cc:374] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.090552 4670 raft_consensus.cc:729] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2725753eee04eb1884ccbb659979cc1, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.092882 4670 consensus_queue.cc:260] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.093340 4694 ts_tablet_manager.cc:1918] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:26.093529 4694 log.cc:1196] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/9673d79ba3e24753bc71f1f6d241745b
I20260927 15:19:26.096750 4670 ts_tablet_manager.cc:1428] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260927 15:19:26.093420 4668 raft_consensus.cc:374] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.097131 4668 raft_consensus.cc:729] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f87eca6cc464749872737ae59599c34, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.097637 4668 consensus_queue.cc:260] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.098650 4668 ts_tablet_manager.cc:1428] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20260927 15:19:26.103359 4693 ts_tablet_manager.cc:1918] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:26.103977 4693 log.cc:1196] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/9673d79ba3e24753bc71f1f6d241745b
I20260927 15:19:26.104681 4693 ts_tablet_manager.cc:1939] T 9673d79ba3e24753bc71f1f6d241745b P eafe060205014c3d9e11cffdfe583f84: Deleting consensus metadata
I20260927 15:19:26.108424 4695 ts_tablet_manager.cc:1918] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:26.108556 4671 tablet_bootstrap.cc:492] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: No bootstrap required, opened a new log
I20260927 15:19:26.108759 4695 log.cc:1196] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/9673d79ba3e24753bc71f1f6d241745b
I20260927 15:19:26.108979 4671 ts_tablet_manager.cc:1397] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Time spent bootstrapping tablet: real 0.051s user 0.022s sys 0.023s
I20260927 15:19:26.109921 4695 ts_tablet_manager.cc:1939] T 9673d79ba3e24753bc71f1f6d241745b P 1f87eca6cc464749872737ae59599c34: Deleting consensus metadata
I20260927 15:19:26.110242 4671 raft_consensus.cc:348] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.110577 4671 raft_consensus.cc:374] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.110711 4671 raft_consensus.cc:729] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafe060205014c3d9e11cffdfe583f84, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.111085 4671 consensus_queue.cc:260] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.111488 4694 ts_tablet_manager.cc:1939] T 9673d79ba3e24753bc71f1f6d241745b P d2725753eee04eb1884ccbb659979cc1: Deleting consensus metadata
I20260927 15:19:26.111898 4671 ts_tablet_manager.cc:1428] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:26.112527 4149 catalog_manager.cc:4841] TS 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549): tablet 9673d79ba3e24753bc71f1f6d241745b (table test-table [id=b8251400bb1447bc94a45121fb785ba4]) successfully deleted
I20260927 15:19:26.113758 4146 catalog_manager.cc:4841] TS eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167): tablet 9673d79ba3e24753bc71f1f6d241745b (table test-table [id=b8251400bb1447bc94a45121fb785ba4]) successfully deleted
I20260927 15:19:26.113826 4148 catalog_manager.cc:4841] TS d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183): tablet 9673d79ba3e24753bc71f1f6d241745b (table test-table [id=b8251400bb1447bc94a45121fb785ba4]) successfully deleted
I20260927 15:19:26.116245 4676 raft_consensus.cc:482] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:26.116511 4676 raft_consensus.cc:504] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.120653 4676 leader_election.cc:288] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183), 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549)
I20260927 15:19:26.130576 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.120155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:44487 (local address 127.2.246.66:41549)
0927 15:19:26.120438 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.120447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.122511 (+ 2064us) server_negotiation.cc:407] Connection header received
0927 15:19:26.122896 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.122902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.123779 (+ 877us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.123905 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:26.124667 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.125701 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.128042 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.129460 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.129664 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.129748 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.130134 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.130273 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:19:26.130692 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.119550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41549 (local address 127.2.246.67:44487)
0927 15:19:26.122366 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.122392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.122643 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.123971 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.123977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.124009 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:26.124450 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.124464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.125891 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.125900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.127742 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.127758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.129684 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.129691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.129941 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.129984 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.130102 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.130281 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2572,"mutex_wait_us":62,"thread_start_us":1769,"threads_started":1}
I20260927 15:19:26.131971 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.119403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:38183 (local address 127.2.246.67:38545)
0927 15:19:26.121826 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.121895 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.122196 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.122604 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.122609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.122657 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:26.123850 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.123867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.125278 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.125284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.127857 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.127931 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.131077 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.131086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.131279 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.131315 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.131475 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.131614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2205,"mutex_wait_us":39,"spinlock_wait_cycles":2944,"thread_start_us":2510,"threads_started":1}
I20260927 15:19:26.132071 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.119461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38545 (local address 127.2.246.65:38183)
0927 15:19:26.119756 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.119767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.122044 (+ 2277us) server_negotiation.cc:407] Connection header received
0927 15:19:26.122370 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.122376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.122459 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.122573 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:26.124077 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.125118 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.128124 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.129461 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.129657 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.129748 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.131572 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.131758 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 15:19:26.131875 4459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0556b3f9794da780de073b6bd3f59a" candidate_uuid: "eafe060205014c3d9e11cffdfe583f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f87eca6cc464749872737ae59599c34" is_pre_election: true
I20260927 15:19:26.132475 4459 raft_consensus.cc:2454] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafe060205014c3d9e11cffdfe583f84 in term 0.
I20260927 15:19:26.133208 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0556b3f9794da780de073b6bd3f59a" candidate_uuid: "eafe060205014c3d9e11cffdfe583f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2725753eee04eb1884ccbb659979cc1" is_pre_election: true
I20260927 15:19:26.133308 4532 leader_election.cc:302] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f87eca6cc464749872737ae59599c34, eafe060205014c3d9e11cffdfe583f84; no voters:
I20260927 15:19:26.133512 4323 raft_consensus.cc:2454] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafe060205014c3d9e11cffdfe583f84 in term 0.
I20260927 15:19:26.134009 4676 raft_consensus.cc:2781] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:26.134140 4676 raft_consensus.cc:482] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:26.134250 4676 raft_consensus.cc:3037] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.137971 4676 raft_consensus.cc:504] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.138954 4676 leader_election.cc:288] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [CANDIDATE]: Term 1 election: Requested vote from peers d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183), 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549)
I20260927 15:19:26.139559 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0556b3f9794da780de073b6bd3f59a" candidate_uuid: "eafe060205014c3d9e11cffdfe583f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2725753eee04eb1884ccbb659979cc1"
I20260927 15:19:26.139850 4323 raft_consensus.cc:3037] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.139791 4459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0556b3f9794da780de073b6bd3f59a" candidate_uuid: "eafe060205014c3d9e11cffdfe583f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f87eca6cc464749872737ae59599c34"
I20260927 15:19:26.140136 4459 raft_consensus.cc:3037] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.142652 4323 raft_consensus.cc:2454] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafe060205014c3d9e11cffdfe583f84 in term 1.
I20260927 15:19:26.143374 4531 leader_election.cc:302] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2725753eee04eb1884ccbb659979cc1, eafe060205014c3d9e11cffdfe583f84; no voters:
I20260927 15:19:26.143818 4676 raft_consensus.cc:2781] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:26.143862 4459 raft_consensus.cc:2454] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafe060205014c3d9e11cffdfe583f84 in term 1.
I20260927 15:19:26.144428 4676 raft_consensus.cc:686] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 1 LEADER]: Becoming Leader. State: Replica: eafe060205014c3d9e11cffdfe583f84, State: Running, Role: LEADER
I20260927 15:19:26.145155 4676 consensus_queue.cc:237] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.151046 4162 catalog_manager.cc:5495] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafe060205014c3d9e11cffdfe583f84 (127.2.246.67). New cstate: current_term: 1 leader_uuid: "eafe060205014c3d9e11cffdfe583f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } health_report { overall_health: HEALTHY } } }
Sep 27 15:19:26 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522363, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:26.228709 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.209204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:36439 (local address 127.0.0.1:50122)
0927 15:19:26.210088 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.210124 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.210348 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.210698 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.210703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.212296 (+ 1593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:26.212545 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.212557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.213735 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.213741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.214816 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.214827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.215634 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.215639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.215807 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.216709 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:26.216742 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:26.220195 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:26.224268 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:26.224290 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:26.224302 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:26.225086 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:26.225674 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:26.225680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:26.225685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:26.225943 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:26.226707 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:26.226718 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:26.227008 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.227259 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.227651 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":435,"threads_started":1}
I20260927 15:19:26.228732 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.209285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.2.246.126:36439)
0927 15:19:26.210394 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.210402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.210431 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:19:26.210490 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.210495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.210551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.210677 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.212761 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.213585 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.214965 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.215420 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.215575 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.220682 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.220692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.220699 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.220756 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.224015 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.225303 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.225308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.225310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.225501 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.226143 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.226148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.226151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.226530 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.226714 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.227444 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.227849 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":986,"thread_start_us":812,"threads_started":1}
I20260927 15:19:26.236919 4162 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:50122:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:19:26.238561 4162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:26.248318 4298 tablet_service.cc:1467] Processing CreateTablet for tablet aa6cf2dd64564d70b3dfde2e79a30a88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fe6f5fc870e43a49318e4ab0f6f4c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:26.248631 4438 tablet_service.cc:1467] Processing CreateTablet for tablet aa6cf2dd64564d70b3dfde2e79a30a88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fe6f5fc870e43a49318e4ab0f6f4c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:26.249121 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6cf2dd64564d70b3dfde2e79a30a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.249325 4438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6cf2dd64564d70b3dfde2e79a30a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.255846 4670 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: Bootstrap starting.
I20260927 15:19:26.257784 4579 tablet_service.cc:1467] Processing CreateTablet for tablet aa6cf2dd64564d70b3dfde2e79a30a88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fe6f5fc870e43a49318e4ab0f6f4c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:26.258270 4670 tablet_bootstrap.cc:654] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.258505 4579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6cf2dd64564d70b3dfde2e79a30a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:26.260673 4670 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: No bootstrap required, opened a new log
I20260927 15:19:26.260922 4670 ts_tablet_manager.cc:1397] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 15:19:26.261703 4670 raft_consensus.cc:348] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.262050 4670 raft_consensus.cc:374] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.262137 4670 raft_consensus.cc:729] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2725753eee04eb1884ccbb659979cc1, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.262521 4670 consensus_queue.cc:260] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.263082 4675 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.263563 4675 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.266063 4668 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: Bootstrap starting.
I20260927 15:19:26.267445 4671 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: Bootstrap starting.
I20260927 15:19:26.268034 4670 ts_tablet_manager.cc:1428] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 15:19:26.268224 4668 tablet_bootstrap.cc:654] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.269702 4671 tablet_bootstrap.cc:654] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:26.275187 4671 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: No bootstrap required, opened a new log
I20260927 15:19:26.275417 4671 ts_tablet_manager.cc:1397] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 15:19:26.276285 4671 raft_consensus.cc:348] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.276551 4671 raft_consensus.cc:374] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.276651 4671 raft_consensus.cc:729] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafe060205014c3d9e11cffdfe583f84, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.276942 4671 consensus_queue.cc:260] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.277390 4676 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.277596 4671 ts_tablet_manager.cc:1428] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:26.277760 4676 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.278668 4668 tablet_bootstrap.cc:492] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: No bootstrap required, opened a new log
I20260927 15:19:26.278888 4668 ts_tablet_manager.cc:1397] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260927 15:19:26.280109 4668 raft_consensus.cc:348] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.280422 4668 raft_consensus.cc:374] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:26.280531 4668 raft_consensus.cc:729] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f87eca6cc464749872737ae59599c34, State: Initialized, Role: FOLLOWER
I20260927 15:19:26.280875 4668 consensus_queue.cc:260] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.281999 4668 ts_tablet_manager.cc:1428] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:26.281775 4689 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.283124 4689 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.404193 4675 raft_consensus.cc:482] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:26.404537 4675 raft_consensus.cc:504] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.406548 4675 leader_election.cc:288] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549), eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167)
I20260927 15:19:26.426612 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.406878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:46167 (local address 127.2.246.65:48869)
0927 15:19:26.418081 (+ 11203us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.418108 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.418481 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.419168 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.419174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.419205 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:26.420546 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.420564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.422615 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.422621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.424356 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.424368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.425639 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.425645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.425813 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.425920 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.426028 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.426196 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11059,"thread_start_us":10868,"threads_started":1}
I20260927 15:19:26.426774 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.407354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:48869 (local address 127.2.246.67:46167)
0927 15:19:26.417732 (+ 10378us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.417742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.418268 (+ 526us) server_negotiation.cc:407] Connection header received
0927 15:19:26.418735 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.418741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.418847 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.418933 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:26.420913 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.422415 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.424535 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.425380 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.425593 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.425683 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.426093 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.426299 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10208,"thread_start_us":10040,"threads_started":1}
I20260927 15:19:26.428051 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6cf2dd64564d70b3dfde2e79a30a88" candidate_uuid: "d2725753eee04eb1884ccbb659979cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafe060205014c3d9e11cffdfe583f84" is_pre_election: true
I20260927 15:19:26.428411 4599 raft_consensus.cc:2454] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2725753eee04eb1884ccbb659979cc1 in term 0.
I20260927 15:19:26.428722 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.407864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:54997 (local address 127.2.246.66:41549)
0927 15:19:26.408128 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.408138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.418748 (+ 10610us) server_negotiation.cc:407] Connection header received
0927 15:19:26.420956 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.420964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.421104 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.421207 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:26.421815 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.422753 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.425971 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.427346 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.427561 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.427648 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.428185 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.428349 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:19:26.429441 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.407051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41549 (local address 127.2.246.65:54997)
0927 15:19:26.418649 (+ 11598us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.418669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.418847 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.421283 (+ 2436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.421288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.421314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:26.421657 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.421665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.422959 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.422972 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.425688 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.425717 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.427804 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.427810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.427982 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.428021 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.428671 (+ 650us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.428858 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11426,"thread_start_us":11207,"threads_started":1}
I20260927 15:19:26.429574 4249 leader_election.cc:302] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2725753eee04eb1884ccbb659979cc1, eafe060205014c3d9e11cffdfe583f84; no voters:
I20260927 15:19:26.430238 4459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6cf2dd64564d70b3dfde2e79a30a88" candidate_uuid: "d2725753eee04eb1884ccbb659979cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f87eca6cc464749872737ae59599c34" is_pre_election: true
I20260927 15:19:26.430397 4675 raft_consensus.cc:2781] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:26.430526 4675 raft_consensus.cc:482] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:26.430563 4459 raft_consensus.cc:2454] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2725753eee04eb1884ccbb659979cc1 in term 0.
I20260927 15:19:26.430620 4675 raft_consensus.cc:3037] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.433331 4675 raft_consensus.cc:504] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.434221 4675 leader_election.cc:288] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [CANDIDATE]: Term 1 election: Requested vote from peers 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549), eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167)
I20260927 15:19:26.434926 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6cf2dd64564d70b3dfde2e79a30a88" candidate_uuid: "d2725753eee04eb1884ccbb659979cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafe060205014c3d9e11cffdfe583f84"
I20260927 15:19:26.435243 4599 raft_consensus.cc:3037] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.435308 4459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6cf2dd64564d70b3dfde2e79a30a88" candidate_uuid: "d2725753eee04eb1884ccbb659979cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f87eca6cc464749872737ae59599c34"
I20260927 15:19:26.435663 4459 raft_consensus.cc:3037] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:26.438741 4459 raft_consensus.cc:2454] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2725753eee04eb1884ccbb659979cc1 in term 1.
I20260927 15:19:26.439190 4599 raft_consensus.cc:2454] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2725753eee04eb1884ccbb659979cc1 in term 1.
I20260927 15:19:26.439450 4252 leader_election.cc:302] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f87eca6cc464749872737ae59599c34, d2725753eee04eb1884ccbb659979cc1; no voters:
I20260927 15:19:26.439848 4675 raft_consensus.cc:2781] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:26.440462 4675 raft_consensus.cc:686] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [term 1 LEADER]: Becoming Leader. State: Replica: d2725753eee04eb1884ccbb659979cc1, State: Running, Role: LEADER
I20260927 15:19:26.440922 4675 consensus_queue.cc:237] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } }
I20260927 15:19:26.443481 4716 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: TxnStatusTablet state changed. Reason: New leader d2725753eee04eb1884ccbb659979cc1. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.444053 4716 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:26.445351 4718 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:19:26.445624 4718 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:19:26.445772 4161 catalog_manager.cc:5495] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2725753eee04eb1884ccbb659979cc1 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:26.494320 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.484341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:38183 (local address 127.0.0.1:40034)
0927 15:19:26.485021 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.485048 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.485264 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.486237 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.486243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.486260 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:26.486484 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.486497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.488315 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.488322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.490192 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.490209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.491916 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.491925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.492142 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.492233 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.493589 (+ 1356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.493611 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.493708 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.493931 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462}
I20260927 15:19:26.495205 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.484712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.2.246.65:38183)
0927 15:19:26.485019 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.485026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.485171 (+ 145us) server_negotiation.cc:407] Connection header received
0927 15:19:26.485557 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.485564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.485740 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.486007 (+ 267us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:26.486676 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.488067 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.490797 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.491588 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.491733 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.492504 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.493358 (+ 854us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.493521 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.494734 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.494849 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:19:26.502951 4599 raft_consensus.cc:1261] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Refusing update from remote peer d2725753eee04eb1884ccbb659979cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:26.503041 4459 raft_consensus.cc:1261] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Refusing update from remote peer d2725753eee04eb1884ccbb659979cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:26.504179 4675 consensus_queue.cc:1035] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:26.504953 4675 consensus_queue.cc:1035] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:26.520711 4689 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: TxnStatusTablet state changed. Reason: New leader d2725753eee04eb1884ccbb659979cc1. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.521111 4689 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.526805 4716 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.527084 4716 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:26.542495 4701 consensus_queue.cc:1035] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:26.544413 4689 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.544746 4689 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P 1f87eca6cc464749872737ae59599c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.565953 4676 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: TxnStatusTablet state changed. Reason: New leader d2725753eee04eb1884ccbb659979cc1. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.566357 4676 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.566550 4675 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.566995 4675 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P d2725753eee04eb1884ccbb659979cc1: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:26.571141 4722 mvcc.cc:205] Tried to move back new op lower bound from 7333979613184421888 to 7333979612946808832. Current Snapshot: MvccSnapshot[applied={T|T < 7333979613184421888}]
I20260927 15:19:26.581112 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.552066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.2.246.67:46167)
0927 15:19:26.552327 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.552334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.572192 (+ 19858us) server_negotiation.cc:407] Connection header received
0927 15:19:26.572806 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.572813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.572958 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.573105 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:26.573912 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.575197 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.577994 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.578676 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.578788 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.579471 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.580228 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.580355 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.580588 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.580695 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:19:26.604202 4701 consensus_queue.cc:1035] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:26.605525 4676 tablet_replica.cc:439] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2725753eee04eb1884ccbb659979cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2725753eee04eb1884ccbb659979cc1" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 38183 } } peers { permanent_uuid: "1f87eca6cc464749872737ae59599c34" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41549 } } peers { permanent_uuid: "eafe060205014c3d9e11cffdfe583f84" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 46167 } } }
I20260927 15:19:26.605894 4676 tablet_replica.cc:442] T aa6cf2dd64564d70b3dfde2e79a30a88 P eafe060205014c3d9e11cffdfe583f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:26.632099 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.623030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:49815 (local address 127.2.246.65:38183)
0927 15:19:26.623317 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.623327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.624455 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 15:19:26.624752 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.624757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.624848 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.624912 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:26.625698 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.626922 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.629326 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.630106 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.630255 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.630833 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.630985 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.631082 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.631250 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.631334 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:19:26.632262 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.622506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:38183 (local address 127.2.246.67:49815)
0927 15:19:26.624354 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.624375 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.624577 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.625098 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.625103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.625130 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:26.625475 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.625491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.627238 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.627246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.629121 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.629141 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.630340 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.630347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.630538 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.630626 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.631102 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.631112 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.631200 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.631351 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"thread_start_us":1474,"threads_started":1}
I20260927 15:19:26.689610 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.681233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:43089 (local address 127.2.246.67:46167)
0927 15:19:26.681450 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.681457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.682840 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 15:19:26.683061 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.683067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.683244 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.683361 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:26.683976 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.684808 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.687175 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.687914 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.688097 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.688705 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.688847 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.688947 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.689113 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.689203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:19:26.689996 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.681642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:46167 (local address 127.2.246.65:43089)
0927 15:19:26.682696 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.682718 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:26.682876 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:26.683384 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:26.683389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:26.683404 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:26.683782 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.683794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.685018 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.685024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.686987 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:26.687005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.688192 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.688199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:26.688420 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.688500 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:26.688973 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:26.688985 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:19:26.689096 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:19:26.689266 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":850,"threads_started":1}
I20260927 15:19:26.752545 4161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:26.752825 4161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:26.756953 4161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2ed80c8895840b396bc93b0e2bfc4a2: Sending DeleteTablet for 3 replicas of tablet ae0556b3f9794da780de073b6bd3f59a
I20260927 15:19:26.758266 4298 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0556b3f9794da780de073b6bd3f59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:40018
I20260927 15:19:26.758343 4579 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0556b3f9794da780de073b6bd3f59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260927 15:19:26.759233 4438 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0556b3f9794da780de073b6bd3f59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:26 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:39736
I20260927 15:19:26.786628 4746 tablet_replica.cc:330] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: stopping tablet replica
I20260927 15:19:26.787396 4746 raft_consensus.cc:2229] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:26.788450 4746 raft_consensus.cc:2258] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.790282 4745 tablet_replica.cc:330] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: stopping tablet replica
I20260927 15:19:26.793270 4746 ts_tablet_manager.cc:1905] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.793648 4745 raft_consensus.cc:2229] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:26.794117 4745 raft_consensus.cc:2258] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.800173 4744 tablet_replica.cc:330] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: stopping tablet replica
I20260927 15:19:26.800376 4745 ts_tablet_manager.cc:1905] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.801296 4744 raft_consensus.cc:2229] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:26.802169 4744 raft_consensus.cc:2258] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:26.807375 4744 ts_tablet_manager.cc:1905] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:26.809306 4745 ts_tablet_manager.cc:1918] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:26.809469 4745 log.cc:1196] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/ae0556b3f9794da780de073b6bd3f59a
I20260927 15:19:26.810007 4745 ts_tablet_manager.cc:1939] T ae0556b3f9794da780de073b6bd3f59a P d2725753eee04eb1884ccbb659979cc1: Deleting consensus metadata
I20260927 15:19:26.811297 4148 catalog_manager.cc:4841] TS d2725753eee04eb1884ccbb659979cc1 (127.2.246.65:38183): tablet ae0556b3f9794da780de073b6bd3f59a (table test-table [id=595e1a7d34774b868830fe742cc1fbc2]) successfully deleted
I20260927 15:19:26.816627 4746 ts_tablet_manager.cc:1918] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:26.816812 4746 log.cc:1196] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/ae0556b3f9794da780de073b6bd3f59a
I20260927 15:19:26.817684 4746 ts_tablet_manager.cc:1939] T ae0556b3f9794da780de073b6bd3f59a P 1f87eca6cc464749872737ae59599c34: Deleting consensus metadata
I20260927 15:19:26.819120 4149 catalog_manager.cc:4841] TS 1f87eca6cc464749872737ae59599c34 (127.2.246.66:41549): tablet ae0556b3f9794da780de073b6bd3f59a (table test-table [id=595e1a7d34774b868830fe742cc1fbc2]) successfully deleted
Sep 27 15:19:26 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:26.823032 4744 ts_tablet_manager.cc:1918] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:26.823251 4744 log.cc:1196] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/ae0556b3f9794da780de073b6bd3f59a
I20260927 15:19:26.823770 4744 ts_tablet_manager.cc:1939] T ae0556b3f9794da780de073b6bd3f59a P eafe060205014c3d9e11cffdfe583f84: Deleting consensus metadata
I20260927 15:19:26.825234 4146 catalog_manager.cc:4841] TS eafe060205014c3d9e11cffdfe583f84 (127.2.246.67:46167): tablet ae0556b3f9794da780de073b6bd3f59a (table test-table [id=595e1a7d34774b868830fe742cc1fbc2]) successfully deleted
I20260927 15:19:26.833149 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.2.246.65:38183)
0927 15:19:26.778257 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.778265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.788176 (+ 9911us) server_negotiation.cc:407] Connection header received
0927 15:19:26.790931 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.790939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.791057 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.791153 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.802437 (+ 11284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.803787 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.809715 (+ 5928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.810496 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.810626 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.824866 (+ 14240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.824888 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.824918 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.824995 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.829416 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.830406 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.830415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.830418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.830573 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.830964 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.830969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.830972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.831370 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.831752 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.832284 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.832569 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 15:19:26.834508 4364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40038
I20260927 15:19:26.858603 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.843246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.2.246.126:36439)
0927 15:19:26.843917 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.843925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.844574 (+ 649us) server_negotiation.cc:407] Connection header received
0927 15:19:26.844818 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.844824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.844903 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.845039 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.846557 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.847621 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.849106 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.849577 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.849717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.852394 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.852408 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.852412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.852455 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.855427 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.856189 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.856195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.856199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.856344 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.856723 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.856730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.856733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.857033 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.857301 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.857623 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.857929 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":328,"threads_started":1}
W20260927 15:19:26.859465 4161 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50132
I20260927 15:19:26.862470 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4233
I20260927 15:19:26.890053 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.863898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.2.246.65:38183)
0927 15:19:26.866664 (+ 2766us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.866675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.874638 (+ 7963us) server_negotiation.cc:407] Connection header received
0927 15:19:26.874892 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.874897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.874983 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.875110 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.876437 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.877307 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.878687 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.879240 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.879397 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.881983 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.881993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.881998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.882053 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.886445 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.887464 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.887472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.887476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.887636 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.888089 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.888094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.888098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.888450 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.888786 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.889126 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.889390 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1912}
W20260927 15:19:26.890857 4364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40048
W20260927 15:19:26.891635 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4233: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:26.891860 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4233
I20260927 15:19:26.920578 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4373
Sep 27 15:19:26 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:26.941123 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.2.246.66:41549)
0927 15:19:26.921770 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.921777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.921803 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:19:26.922061 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.922067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.922175 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.922311 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.923605 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.924717 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.926657 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.927451 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.927616 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.933060 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.933080 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.933106 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.933156 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.937518 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.938589 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.938599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.938603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.938796 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.939223 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.939230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.939234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.939715 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.940057 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.940403 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.940666 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 15:19:26.942199 4504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39756
W20260927 15:19:26.943128 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4373: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:26.943392 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4373
I20260927 15:19:26.969238 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4513
Sep 27 15:19:26 dist-test-slave-14jq krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:26.990995 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.970408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.2.246.67:46167)
0927 15:19:26.970669 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:26.970679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:26.970709 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:19:26.970858 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:26.970864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:26.970985 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:26.971111 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:26.972580 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.973934 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.975311 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:26.976103 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:26.976284 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:26.981709 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:26.981733 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:26.981767 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:26.981924 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:26.986939 (+ 5015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.988056 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.988067 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.988072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.988261 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:26.988752 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:26.988758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:26.988762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:26.989321 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:26.989655 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:26.990195 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 15:19:26.990477 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 15:19:26.992012 4644 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54044
W20260927 15:19:26.993012 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4513: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:26.993227 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4513
I20260927 15:19:27.015239 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4130
I20260927 15:19:27.016644 4202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:27.155025 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4130
2026-09-27T15:19:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5419 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:27 dist-test-slave-14jq krb5kdc[4756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:27 dist-test-slave-14jq krb5kdc[4756](info): set up 2 sockets
Sep 27 15:19:27 dist-test-slave-14jq krb5kdc[4756](info): commencing operation
krb5kdc: starting...
W20260927 15:19:28.322521 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:28 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:28.544512 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44617
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:28.960672 4769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:28.961174 4769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:28.961350 4769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:28.970414 4769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:28.970626 4769 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:19:28.970732 4769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:28.970865 4769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:28.970968 4769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:28.971098 4769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:28.983388 4769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44617
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:28.985199 4769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:28.986908 4769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:29.006150 4777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:29.006379 4774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:29.006747 4775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:29.007620 4769 server_base.cc:1034] running on GCE node
I20260927 15:19:29.008817 4769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:29.010839 4769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:29.012194 4769 hybrid_clock.cc:648] HybridClock initialized: now 1790522369012104 us; error 61 us; skew 500 ppm
I20260927 15:19:29.012674 4769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:29 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:29.016783 4769 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:29.018950 4769 webserver.cc:460] Webserver started at http://127.2.246.126:42377/ using document root <none> and password file <none>
I20260927 15:19:29.019884 4769 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:29.020035 4769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:29.020401 4769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:29.023929 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "233cc872a7264bb9b9b2975495f5abc5"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "4dd2f30c59bfb01ee25376856bbf50dc"
server_key_iv: "f0085d6b46d37df1e1cc84aea3bf0f36"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.024822 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "233cc872a7264bb9b9b2975495f5abc5"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "4dd2f30c59bfb01ee25376856bbf50dc"
server_key_iv: "f0085d6b46d37df1e1cc84aea3bf0f36"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.031519 4769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:19:29.036212 4784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:29.038187 4769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:29.038453 4769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "233cc872a7264bb9b9b2975495f5abc5"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "4dd2f30c59bfb01ee25376856bbf50dc"
server_key_iv: "f0085d6b46d37df1e1cc84aea3bf0f36"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.038733 4769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:29.056280 4769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:29.061131 4769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:29.061430 4769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:29.087885 4769 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44617
I20260927 15:19:29.087913 4845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44617 every 8 connection(s)
I20260927 15:19:29.089403 4769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:29.094547 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:29.099874 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4769
I20260927 15:19:29.100152 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:29.101063 3033 external_mini_cluster.cc:1428] Setting key 67f8d92673959a34c8795caf41957af6
I20260927 15:19:29.112110 4846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Bootstrap starting.
I20260927 15:19:29.116691 4846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Neither blocks nor log segments found. Creating new log.
Sep 27 15:19:29 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:29.118372 4846 log.cc:824] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:29.123659 4846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: No bootstrap required, opened a new log
I20260927 15:19:29.129519 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.104005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.2.246.126:44617)
0927 15:19:29.105051 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:29.105068 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:29.105119 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:19:29.107128 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:29.107158 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:29.107674 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:29.108157 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:29.109660 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.111137 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.112640 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.113426 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.113628 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:29.118296 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:29.118306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:29.118326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:29.118379 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:29.123011 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.124166 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.124179 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.124192 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.124550 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.125301 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.125307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.125311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.126442 (+ 1131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:29.126802 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:29.127235 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:19:29.127841 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":480,"threads_started":1}
I20260927 15:19:29.137607 4846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } }
I20260927 15:19:29.138221 4846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:29.138317 4846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233cc872a7264bb9b9b2975495f5abc5, State: Initialized, Role: FOLLOWER
I20260927 15:19:29.139096 4846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } }
I20260927 15:19:29.139372 4846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:29.139496 4846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:29.139664 4846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:29.142508 4846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } }
I20260927 15:19:29.143373 4846 leader_election.cc:302] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 233cc872a7264bb9b9b2975495f5abc5; no voters:
I20260927 15:19:29.144465 4846 leader_election.cc:288] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:29.144623 4851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:29.145785 4851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [term 1 LEADER]: Becoming Leader. State: Replica: 233cc872a7264bb9b9b2975495f5abc5, State: Running, Role: LEADER
I20260927 15:19:29.146538 4851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } }
I20260927 15:19:29.147451 4846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:29.152137 4853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 233cc872a7264bb9b9b2975495f5abc5. Latest consensus state: current_term: 1 leader_uuid: "233cc872a7264bb9b9b2975495f5abc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } } }
I20260927 15:19:29.152761 4853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:29.154103 4852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "233cc872a7264bb9b9b2975495f5abc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233cc872a7264bb9b9b2975495f5abc5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44617 } } }
I20260927 15:19:29.154697 4863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:29.154914 4852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:29.160907 4863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:29.175174 4863 catalog_manager.cc:1347] Generated new cluster ID: 8cc3c79ffe1c46d189f2adc63fce3fa0
I20260927 15:19:29.175385 4863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:29.200717 4863 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:29.202843 4863 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:29.215896 4863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Generated new TSK 0
I20260927 15:19:29.217228 4863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:29.283522 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44617
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:29.711434 4872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:29.711822 4872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:29.712029 4872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:29.720605 4872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:29.720754 4872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:29.720922 4872 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:19:29.720992 4872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:29.721115 4872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:29.732749 4872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:29.734622 4872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:29.736263 4872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:29.754621 4880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:29.754586 4878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:29.754794 4877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:29.755592 4872 server_base.cc:1034] running on GCE node
I20260927 15:19:29.756428 4872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:29.757732 4872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:29.758998 4872 hybrid_clock.cc:648] HybridClock initialized: now 1790522369758925 us; error 58 us; skew 500 ppm
I20260927 15:19:29.759421 4872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:29 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:29.763288 4872 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:29.765542 4872 webserver.cc:460] Webserver started at http://127.2.246.65:39037/ using document root <none> and password file <none>
I20260927 15:19:29.766539 4872 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:29.766690 4872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:29.767028 4872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:29.770773 4872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "7c6de3cf43034b71abcccd96d85b3f42"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "c1f9bf627a49fe0e3997c2d65ee02bb5"
server_key_iv: "b9acfa1af87799ddc18d2b97302d1f79"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.771651 4872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "7c6de3cf43034b71abcccd96d85b3f42"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "c1f9bf627a49fe0e3997c2d65ee02bb5"
server_key_iv: "b9acfa1af87799ddc18d2b97302d1f79"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.778266 4872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:19:29.782943 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:29.785003 4872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:29.785283 4872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "7c6de3cf43034b71abcccd96d85b3f42"
format_stamp: "Formatted at 2026-09-27 15:19:29 on dist-test-slave-14jq"
server_key: "c1f9bf627a49fe0e3997c2d65ee02bb5"
server_key_iv: "b9acfa1af87799ddc18d2b97302d1f79"
server_key_version: "encryptionkey@0"
I20260927 15:19:29.785573 4872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:29.801689 4872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:29.807257 4872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:29.807647 4872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:29.811324 4872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:29.811512 4872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:29.811667 4872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:29.811811 4872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:29 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:29.841425 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.812030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:43275 (local address 127.2.246.126:44617)
0927 15:19:29.812943 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:29.812951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:29.814262 (+ 1311us) server_negotiation.cc:407] Connection header received
0927 15:19:29.816871 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:29.816881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:29.817034 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:29.817266 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:29.821043 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.822622 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.825622 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.826141 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.826306 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:29.832227 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:29.832240 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:29.832249 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:29.832316 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:29.836640 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.838327 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.838337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.838343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.838541 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.839152 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.839157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.839168 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.839708 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:29.839893 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:29.840748 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 15:19:29.841002 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":616,"threads_started":1}
I20260927 15:19:29.842692 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.812924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.65:43275)
0927 15:19:29.814039 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:29.814091 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:29.816468 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:29.817602 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:29.817617 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:29.818548 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:29.820746 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:29.820781 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.822883 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.822894 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:29.825420 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:29.825434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.826513 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.826522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:29.826765 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:29.827736 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:29.827759 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:29.831728 (+ 3969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:29.836912 (+ 5184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:29.836932 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:29.836979 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:29.838054 (+ 1075us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:29.838739 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:29.838744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:29.838747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:29.839016 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:29.839900 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:29.839912 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:29.840342 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 15:19:29.840637 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 15:19:29.841269 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":469,"threads_started":1}
I20260927 15:19:29.877662 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.853689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.65:37053)
0927 15:19:29.859047 (+ 5358us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:29.859072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:29.859290 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:29.859933 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:29.859937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:29.860418 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:29.860729 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:29.860738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.862635 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.862646 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:29.863748 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:29.863763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.865562 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.865573 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:29.865827 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:29.866655 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:29.866685 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:29.867962 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:29.873213 (+ 5251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:29.873225 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:29.873233 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:29.873809 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:29.874537 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:29.874545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:29.874549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:29.874781 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:29.875685 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:29.875694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:29.876028 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 15:19:29.876224 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 15:19:29.876938 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5180,"thread_start_us":5004,"threads_started":1}
I20260927 15:19:29.877885 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.853497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:37053 (local address 127.2.246.126:44617)
0927 15:19:29.853783 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:29.853791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:29.859248 (+ 5457us) server_negotiation.cc:407] Connection header received
0927 15:19:29.859501 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:29.859512 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:29.859667 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:29.859870 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:29.860916 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.862420 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.863946 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:29.865238 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:29.865481 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:29.868444 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:29.868457 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:29.868466 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:29.868532 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:29.872921 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.874094 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.874101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.874106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.874314 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:29.874984 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:29.874992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:29.874998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:29.875505 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:29.876862 (+ 1357us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:29.877045 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 15:19:29.877373 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:19:29.880954 5007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:33555 every 8 connection(s)
I20260927 15:19:29.881117 4872 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:33555
I20260927 15:19:29.882740 4872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:29.885283 5008 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44617
I20260927 15:19:29.885578 5008 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:29.886439 5008 heartbeater.cc:511] Master 127.2.246.126:44617 requested a full tablet report, sending...
I20260927 15:19:29.888878 4801 ts_manager.cc:194] Registered new tserver with Master: 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555)
I20260927 15:19:29.890731 4801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:43275
I20260927 15:19:29.891834 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 4872
I20260927 15:19:29.892338 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:29.893160 3033 external_mini_cluster.cc:1428] Setting key ebd395485063d42413bde8fc74ca019f
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:29.942121 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44617
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:30.389189 5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:30.389612 5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:30.389889 5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:30.400146 5012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:30.400301 5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:30.400403 5012 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:19:30.400465 5012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:30.400542 5012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:30.413031 5012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:30.414789 5012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:30.417106 5012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:30.434100 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:30.434026 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:30.434060 5020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:30.436002 5012 server_base.cc:1034] running on GCE node
I20260927 15:19:30.436905 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:30.438181 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:30.439538 5012 hybrid_clock.cc:648] HybridClock initialized: now 1790522370439446 us; error 87 us; skew 500 ppm
I20260927 15:19:30.440001 5012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:30 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522370, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:30.444284 5012 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:30.446600 5012 webserver.cc:460] Webserver started at http://127.2.246.66:34845/ using document root <none> and password file <none>
I20260927 15:19:30.447536 5012 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:30.447706 5012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:30.448108 5012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:30.451900 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "b5aa60e529054dabb988486b59675e75"
format_stamp: "Formatted at 2026-09-27 15:19:30 on dist-test-slave-14jq"
server_key: "f0211bbd78d479c54d9cfddf161f3332"
server_key_iv: "e904124b29a5df54afa3f2271b37dea1"
server_key_version: "encryptionkey@0"
I20260927 15:19:30.452883 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "b5aa60e529054dabb988486b59675e75"
format_stamp: "Formatted at 2026-09-27 15:19:30 on dist-test-slave-14jq"
server_key: "f0211bbd78d479c54d9cfddf161f3332"
server_key_iv: "e904124b29a5df54afa3f2271b37dea1"
server_key_version: "encryptionkey@0"
I20260927 15:19:30.459266 5012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 15:19:30.463770 5027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:30.465698 5012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:30.465972 5012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "b5aa60e529054dabb988486b59675e75"
format_stamp: "Formatted at 2026-09-27 15:19:30 on dist-test-slave-14jq"
server_key: "f0211bbd78d479c54d9cfddf161f3332"
server_key_iv: "e904124b29a5df54afa3f2271b37dea1"
server_key_version: "encryptionkey@0"
I20260927 15:19:30.466207 5012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:30.489223 5012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:30.494629 5012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:30.494987 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:30.498186 5012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:30.498335 5012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:30.498489 5012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:30.498592 5012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:30 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522370, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:30.531287 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.499277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:33015 (local address 127.2.246.126:44617)
0927 15:19:30.501912 (+ 2635us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:30.501922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:30.502650 (+ 728us) server_negotiation.cc:407] Connection header received
0927 15:19:30.506045 (+ 3395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:30.506053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:30.506207 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:30.506364 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:30.508534 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.509438 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.513737 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.514661 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.514909 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:30.521297 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:30.521305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:30.521311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:30.521370 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:30.525587 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:30.527345 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:30.527355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:30.527361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:30.527597 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:30.528460 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:30.528466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:30.528469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:30.528993 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:30.529222 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:30.530581 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0927 15:19:30.530880 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2429,"thread_start_us":2284,"threads_started":1}
I20260927 15:19:30.532439 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.500544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.66:33015)
0927 15:19:30.502342 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:30.502424 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:30.505396 (+ 2972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:30.506697 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:30.506707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:30.507431 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:30.508231 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:30.508257 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.509656 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.509663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:30.513338 (+ 3675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:30.513385 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.515341 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.515355 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:30.515741 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:30.516927 (+ 1186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:30.516985 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:30.520891 (+ 3906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:30.525921 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:30.525944 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:30.525977 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:30.527062 (+ 1085us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:30.527785 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:30.527793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:30.527799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:30.528255 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:30.529189 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:30.529205 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:30.529968 (+ 763us) client_negotiation.cc:769] Sending connection context
0927 15:19:30.530462 (+ 494us) client_negotiation.cc:240] Negotiation successful
0927 15:19:30.531098 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":538,"threads_started":1}
I20260927 15:19:30.559157 5012 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:44299
I20260927 15:19:30.559269 5147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:44299 every 8 connection(s)
I20260927 15:19:30.560847 5012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:30.562901 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5012
I20260927 15:19:30.563120 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:30.563884 3033 external_mini_cluster.cc:1428] Setting key da0b319752fe53ef67b6d7f53c351918
I20260927 15:19:30.564388 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.544242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.66:46235)
0927 15:19:30.545243 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:30.545267 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:30.545474 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:30.545920 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:30.545924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:30.546455 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:30.546843 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:30.546854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.548443 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.548489 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:30.550381 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:30.550404 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.551683 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.551690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:30.551966 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:30.552938 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:30.552978 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:30.554699 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:30.560003 (+ 5304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:30.560010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:30.560016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:30.560637 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:30.561338 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:30.561346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:30.561352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:30.561578 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:30.562832 (+ 1254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:30.562841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:30.563109 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 15:19:30.563278 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:19:30.563730 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":598,"threads_started":1}
I20260927 15:19:30.563829 5148 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44617
I20260927 15:19:30.564761 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.543982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:46235 (local address 127.2.246.126:44617)
0927 15:19:30.544170 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:30.544175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:30.545385 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 15:19:30.545648 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:30.545657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:30.545755 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:30.545892 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:30.547002 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.548174 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.550610 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:30.551419 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:30.551596 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:30.555045 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:30.555056 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:30.555063 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:30.555136 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:30.559783 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:30.560846 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:30.560851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:30.560854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:30.561195 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:30.561743 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:30.561749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:30.561755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:30.562664 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:30.563985 (+ 1321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:30.564131 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:19:30.564424 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:19:30.565454 5148 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:30.566884 5148 heartbeater.cc:511] Master 127.2.246.126:44617 requested a full tablet report, sending...
I20260927 15:19:30.570591 4800 ts_manager.cc:194] Registered new tserver with Master: b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299)
I20260927 15:19:30.571762 4800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:33015
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:30.613482 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44617
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:30.894768 5008 heartbeater.cc:503] Master 127.2.246.126:44617 was elected leader, sending a full tablet report...
W20260927 15:19:30.983296 5152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:30.983598 5152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:30.983762 5152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:30.991748 5152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:30.991876 5152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:30.992017 5152 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:19:30.992091 5152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:19:30.992193 5152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:31.002563 5152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:31.003961 5152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:31.005270 5152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:31.020928 5158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:31.021402 5157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:31.021072 5160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:31.022157 5152 server_base.cc:1034] running on GCE node
I20260927 15:19:31.022820 5152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:31.023936 5152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:31.025173 5152 hybrid_clock.cc:648] HybridClock initialized: now 1790522371025093 us; error 82 us; skew 500 ppm
I20260927 15:19:31.025573 5152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:31 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:31.029119 5152 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:31.030897 5152 webserver.cc:460] Webserver started at http://127.2.246.67:38261/ using document root <none> and password file <none>
I20260927 15:19:31.031605 5152 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:31.031754 5152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:31.032070 5152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:31.035331 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "72f3a19244ff4a80b6058ef23617451f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-14jq"
server_key: "0a12b8fdd7555d34dfd34f4e6fec45d5"
server_key_iv: "4ebb3c4d7ea99e08a6fba759f2507523"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.036087 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "72f3a19244ff4a80b6058ef23617451f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-14jq"
server_key: "0a12b8fdd7555d34dfd34f4e6fec45d5"
server_key_iv: "4ebb3c4d7ea99e08a6fba759f2507523"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.042012 5152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:31.046360 5167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.048239 5152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:31.048529 5152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "72f3a19244ff4a80b6058ef23617451f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-14jq"
server_key: "0a12b8fdd7555d34dfd34f4e6fec45d5"
server_key_iv: "4ebb3c4d7ea99e08a6fba759f2507523"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.048789 5152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:31.077726 5152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:31.082353 5152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:31.082671 5152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:31.086107 5152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:31.086257 5152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.086428 5152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:31.086546 5152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:19:31 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:31.129309 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.087267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:50143 (local address 127.2.246.126:44617)
0927 15:19:31.094940 (+ 7673us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.094949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.097027 (+ 2078us) server_negotiation.cc:407] Connection header received
0927 15:19:31.100260 (+ 3233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.100268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.100412 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.100602 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.104098 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.105580 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.110190 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.111075 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.111339 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.118734 (+ 7395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.118744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.118753 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.118821 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.124244 (+ 5423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.125930 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.125938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.125941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.126110 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.126663 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.126669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.126675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.127168 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.127342 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.128538 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.128857 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7509,"thread_start_us":7350,"threads_started":1}
I20260927 15:19:31.131356 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.088620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.67:50143)
0927 15:19:31.096830 (+ 8210us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.096878 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.099781 (+ 2903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.101004 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.101021 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.102337 (+ 1316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:31.103758 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.103795 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.105815 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.105820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.109891 (+ 4071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.109927 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.111597 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.111617 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.112094 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.113357 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:31.113388 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:31.118228 (+ 4840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:31.124589 (+ 6361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.124611 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.124656 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.125660 (+ 1004us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.126267 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.126280 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.126282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.126517 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.127442 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:31.127460 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:31.128069 (+ 609us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.128438 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.129262 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7838,"thread_start_us":4407,"threads_started":1}
I20260927 15:19:31.144322 5152 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:41629
I20260927 15:19:31.144526 5287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:41629 every 8 connection(s)
I20260927 15:19:31.145988 5152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:31.148058 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5152
I20260927 15:19:31.148331 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:31.148967 3033 external_mini_cluster.cc:1428] Setting key 203892d7fd7f771ef5f9656445c66fff
I20260927 15:19:31.148880 5288 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44617
I20260927 15:19:31.150216 5288 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:31.151203 5288 heartbeater.cc:511] Master 127.2.246.126:44617 requested a full tablet report, sending...
I20260927 15:19:31.153065 4801 ts_manager.cc:194] Registered new tserver with Master: 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:31.154186 4801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:50143
I20260927 15:19:31.161443 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.140125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.2.246.67:56323)
0927 15:19:31.142479 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.142509 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.142769 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.143784 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.143790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.144758 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:31.145164 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.145219 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.146746 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.146761 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.151078 (+ 4317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.151108 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.152735 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.152744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.152912 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.153628 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:31.153645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:31.154630 (+ 985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:31.158733 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.158737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.158741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.159159 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.159748 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.159752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.159754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.159873 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.160543 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:31.160546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:31.160750 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.160877 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.161121 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"thread_start_us":2016,"threads_started":1}
I20260927 15:19:31.162253 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.140050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:56323 (local address 127.2.246.126:44617)
0927 15:19:31.140488 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.140499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.142664 (+ 2165us) server_negotiation.cc:407] Connection header received
0927 15:19:31.142979 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.142987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.143109 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.143246 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.145414 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.146550 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.151491 (+ 4941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.152228 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.152417 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.154920 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.154929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.154935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.154982 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.158550 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.159409 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.159414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.159416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.159589 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.160046 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.160050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.160052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.160383 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.160559 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.161051 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.161932 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260927 15:19:31.168445 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:19:31 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:19:31 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:31.226725 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.210743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.2.246.126:44617)
0927 15:19:31.211049 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.211056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.211371 (+ 315us) server_negotiation.cc:407] Connection header received
0927 15:19:31.211635 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.211639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.211730 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.211851 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.213315 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.214335 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.215735 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.216218 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.216362 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.220733 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.220739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.220743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.220783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.223863 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.224593 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.224601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.224605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.224756 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.225136 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.225143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.225146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.225467 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.225740 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.226115 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.226362 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 15:19:31.233320 4801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:31.242251 4801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:31.275364 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.259890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.2.246.66:44299)
0927 15:19:31.260952 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.260963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.261933 (+ 970us) server_negotiation.cc:407] Connection header received
0927 15:19:31.262388 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.262398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.262757 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.262972 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:31.264166 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.266731 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.271746 (+ 5015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.273500 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.273673 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.273970 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.274719 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.274976 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":704,"threads_started":1}
I20260927 15:19:31.275384 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.259550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44299 (local address 127.0.0.1:56648)
0927 15:19:31.261530 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.261608 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.262054 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.263302 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.263310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.263342 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:31.263882 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.263899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.267030 (+ 3131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.267039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.271358 (+ 4319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.271401 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.274024 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.274032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.274306 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.274366 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.274542 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.274875 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"thread_start_us":859,"threads_started":1}
I20260927 15:19:31.280997 5078 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 (DEFAULT_TABLE table=test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.282193 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.261018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.2.246.67:41629)
0927 15:19:31.273582 (+ 12564us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.273593 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.273636 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:19:31.273765 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.273776 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.274120 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.275391 (+ 1271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:31.276179 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.277512 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.280008 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.281012 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.281183 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.281470 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.281655 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.281826 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12335,"thread_start_us":13000,"threads_started":1}
I20260927 15:19:31.282552 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.260496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:41629 (local address 127.0.0.1:41330)
0927 15:19:31.265681 (+ 5185us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.265715 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.266576 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.274391 (+ 7815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.274396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.274427 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:31.275147 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.275160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.277687 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.277693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.279829 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.279839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.281245 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.281252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.281435 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.281475 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.281609 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.281776 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4946,"thread_start_us":780,"threads_started":1}
I20260927 15:19:31.283087 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.261688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.0.0.1:38294)
0927 15:19:31.272790 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.272820 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.273018 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.274352 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.274356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.274396 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:31.274930 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.274942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.277025 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.277031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.279414 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.279436 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.282137 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.282146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.282371 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.282418 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.282501 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.282687 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10893,"thread_start_us":11431,"threads_started":1}
I20260927 15:19:31.283421 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb7ac5d24e473986702f8abc2de5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.284999 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.262063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.2.246.65:33555)
0927 15:19:31.273785 (+ 11722us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.273794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.273819 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:19:31.273938 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.273945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.274117 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.274375 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:31.275204 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.276832 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.279817 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.281674 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.283470 (+ 1796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.283748 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.284444 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.284666 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11567,"thread_start_us":11448,"threads_started":1}
I20260927 15:19:31.286798 5218 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 (DEFAULT_TABLE table=test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.288300 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb7ac5d24e473986702f8abc2de5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.288861 4938 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 (DEFAULT_TABLE table=test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.290671 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb7ac5d24e473986702f8abc2de5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.303385 5307 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Bootstrap starting.
I20260927 15:19:31.305775 5309 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Bootstrap starting.
I20260927 15:19:31.306980 5308 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Bootstrap starting.
I20260927 15:19:31.308806 5307 tablet_bootstrap.cc:654] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.309682 5309 tablet_bootstrap.cc:654] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.310418 5307 log.cc:824] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:31.310855 5309 log.cc:824] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:31.310966 5308 tablet_bootstrap.cc:654] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.312443 5308 log.cc:824] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:31.314144 5307 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: No bootstrap required, opened a new log
I20260927 15:19:31.314525 5307 ts_tablet_manager.cc:1397] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 15:19:31.315294 5309 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: No bootstrap required, opened a new log
I20260927 15:19:31.315642 5309 ts_tablet_manager.cc:1397] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:19:31.315682 5308 tablet_bootstrap.cc:492] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: No bootstrap required, opened a new log
I20260927 15:19:31.315938 5308 ts_tablet_manager.cc:1397] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 15:19:31.322346 5307 raft_consensus.cc:348] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.322813 5307 raft_consensus.cc:374] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.322903 5307 raft_consensus.cc:729] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa60e529054dabb988486b59675e75, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.323603 5307 consensus_queue.cc:260] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.324935 5148 heartbeater.cc:503] Master 127.2.246.126:44617 was elected leader, sending a full tablet report...
I20260927 15:19:31.325479 5307 ts_tablet_manager.cc:1428] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:19:31.328644 5308 raft_consensus.cc:348] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.329160 5308 raft_consensus.cc:374] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.328929 5309 raft_consensus.cc:348] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.329315 5308 raft_consensus.cc:729] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a19244ff4a80b6058ef23617451f, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.329432 5309 raft_consensus.cc:374] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.329562 5309 raft_consensus.cc:729] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.330240 5309 consensus_queue.cc:260] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.330376 5308 consensus_queue.cc:260] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.332057 5309 ts_tablet_manager.cc:1428] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260927 15:19:31.332623 5288 heartbeater.cc:503] Master 127.2.246.126:44617 was elected leader, sending a full tablet report...
I20260927 15:19:31.334451 5308 ts_tablet_manager.cc:1428] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
W20260927 15:19:31.386057 5009 tablet.cc:2369] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:31.398319 5289 tablet.cc:2369] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:31.442497 5313 raft_consensus.cc:482] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:31.443073 5313 raft_consensus.cc:504] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.445268 5313 leader_election.cc:288] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:31.456116 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.445877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:58425 (local address 127.2.246.65:33555)
0927 15:19:31.446122 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.446129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.447155 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 15:19:31.447444 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.447452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.447528 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.447627 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:31.448544 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.449807 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.453254 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.454753 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.454940 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.455024 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.455632 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.455832 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:19:31.456117 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.446059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:38947 (local address 127.2.246.67:41629)
0927 15:19:31.446326 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.446336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.447200 (+ 864us) server_negotiation.cc:407] Connection header received
0927 15:19:31.447543 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.447550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.447639 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.447753 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:31.448683 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.449911 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.453261 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.454730 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.454940 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.455024 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.455632 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.455814 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 15:19:31.456490 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.445968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:41629 (local address 127.2.246.66:38947)
0927 15:19:31.447046 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.447066 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.447366 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.447768 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.447774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.447842 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:31.448501 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.448514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.450048 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.450052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.453000 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.453021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.455068 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.455073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.455342 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.455421 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.455544 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.455718 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":338,"spinlock_wait_cycles":15872,"thread_start_us":692,"threads_started":1}
I20260927 15:19:31.457050 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.445570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.2.246.66:58425)
0927 15:19:31.447012 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.447034 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.447237 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.447659 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.447662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.447684 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:31.448344 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.448358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.450046 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.450052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.453000 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.453021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.455056 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.455063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.455340 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.455420 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.455544 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.456086 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"mutex_wait_us":185,"spinlock_wait_cycles":5376,"thread_start_us":863,"threads_started":1}
I20260927 15:19:31.458038 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb7ac5d24e473986702f8abc2de5d1" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f" is_pre_election: true
I20260927 15:19:31.458222 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb7ac5d24e473986702f8abc2de5d1" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6de3cf43034b71abcccd96d85b3f42" is_pre_election: true
I20260927 15:19:31.458793 4963 raft_consensus.cc:2454] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 0.
I20260927 15:19:31.458794 5238 raft_consensus.cc:2454] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 0.
I20260927 15:19:31.459709 5030 leader_election.cc:302] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, 7c6de3cf43034b71abcccd96d85b3f42, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:31.460258 5313 raft_consensus.cc:2781] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:31.460421 5313 raft_consensus.cc:482] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:31.460534 5313 raft_consensus.cc:3037] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.463382 5313 raft_consensus.cc:504] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.464195 5313 leader_election.cc:288] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 election: Requested vote from peers 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:31.464884 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb7ac5d24e473986702f8abc2de5d1" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6de3cf43034b71abcccd96d85b3f42"
I20260927 15:19:31.465204 4963 raft_consensus.cc:3037] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.465193 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb7ac5d24e473986702f8abc2de5d1" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f"
I20260927 15:19:31.465464 5238 raft_consensus.cc:3037] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.467935 5238 raft_consensus.cc:2454] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 1.
I20260927 15:19:31.468256 4963 raft_consensus.cc:2454] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 1.
I20260927 15:19:31.468575 5031 leader_election.cc:302] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:31.469033 5313 raft_consensus.cc:2781] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:31.469640 5313 raft_consensus.cc:686] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 1 LEADER]: Becoming Leader. State: Replica: b5aa60e529054dabb988486b59675e75, State: Running, Role: LEADER
I20260927 15:19:31.470290 5313 consensus_queue.cc:237] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.476307 4801 catalog_manager.cc:5495] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5aa60e529054dabb988486b59675e75 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:19:31 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:31.547916 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.529094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44617 (local address 127.0.0.1:37830)
0927 15:19:31.530044 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.530066 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.530264 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.530703 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.530708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.531710 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:31.532123 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.532137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.533323 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.533332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.534996 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.535025 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.536113 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.536123 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.536319 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.537133 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:31.537155 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:31.540080 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:31.544027 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.544046 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.544054 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.544919 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.545513 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.545522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.545527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.545783 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.546787 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:31.546795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:31.546994 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.547151 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.547470 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":490,"threads_started":1}
I20260927 15:19:31.548249 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.529062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.2.246.126:44617)
0927 15:19:31.529345 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.529354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.530240 (+ 886us) server_negotiation.cc:407] Connection header received
0927 15:19:31.530412 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.530416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.530485 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.530587 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.532325 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.533078 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.535227 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.535826 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.535944 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.540345 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.540351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.540354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.540396 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.543811 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.545112 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.545121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.545129 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.545283 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.546086 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.546091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.546094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.546628 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.546760 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.547465 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.547740 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 15:19:31.555037 4801 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:37830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:19:31.556416 4801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:31.565321 4938 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cc0ed38c34394a96e8818c39c8709 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fdbb2c153164977addce15edcd976bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:31.566479 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cc0ed38c34394a96e8818c39c8709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.565779 5218 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cc0ed38c34394a96e8818c39c8709 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fdbb2c153164977addce15edcd976bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:31.566875 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cc0ed38c34394a96e8818c39c8709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.566346 5078 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cc0ed38c34394a96e8818c39c8709 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fdbb2c153164977addce15edcd976bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:19:31.567308 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cc0ed38c34394a96e8818c39c8709. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 15:19:31.569664 5149 tablet.cc:2369] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:31.574939 5309 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: Bootstrap starting.
I20260927 15:19:31.575191 5307 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: Bootstrap starting.
I20260927 15:19:31.576409 5309 tablet_bootstrap.cc:654] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.577132 5307 tablet_bootstrap.cc:654] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.578392 5308 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: Bootstrap starting.
I20260927 15:19:31.578701 5309 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: No bootstrap required, opened a new log
I20260927 15:19:31.578874 5309 ts_tablet_manager.cc:1397] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:31.579681 5309 raft_consensus.cc:348] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.579928 5309 raft_consensus.cc:374] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.580031 5309 raft_consensus.cc:729] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.580217 5308 tablet_bootstrap.cc:654] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.580293 5309 consensus_queue.cc:260] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.581005 5309 ts_tablet_manager.cc:1428] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:19:31.580929 5314 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.581440 5314 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.584002 5307 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: No bootstrap required, opened a new log
I20260927 15:19:31.584213 5307 ts_tablet_manager.cc:1397] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 15:19:31.585158 5307 raft_consensus.cc:348] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.585441 5307 raft_consensus.cc:374] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.585551 5307 raft_consensus.cc:729] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa60e529054dabb988486b59675e75, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.585806 5307 consensus_queue.cc:260] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.586612 5307 ts_tablet_manager.cc:1428] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:31.586519 5313 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.586884 5313 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.594687 5308 tablet_bootstrap.cc:492] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: No bootstrap required, opened a new log
I20260927 15:19:31.594935 5308 ts_tablet_manager.cc:1397] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.013s
I20260927 15:19:31.595901 5308 raft_consensus.cc:348] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.596232 5308 raft_consensus.cc:374] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.596360 5308 raft_consensus.cc:729] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a19244ff4a80b6058ef23617451f, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.596625 5308 consensus_queue.cc:260] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.597280 5308 ts_tablet_manager.cc:1428] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:19:31.597188 5315 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.597584 5315 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.619395 5313 raft_consensus.cc:482] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:31.619738 5313 raft_consensus.cc:504] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.620638 5313 leader_election.cc:288] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:31.621302 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cc0ed38c34394a96e8818c39c8709" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6de3cf43034b71abcccd96d85b3f42" is_pre_election: true
I20260927 15:19:31.621517 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cc0ed38c34394a96e8818c39c8709" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f" is_pre_election: true
I20260927 15:19:31.621616 4963 raft_consensus.cc:2454] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 0.
I20260927 15:19:31.621896 5238 raft_consensus.cc:2454] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 0.
I20260927 15:19:31.622255 5030 leader_election.cc:302] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6de3cf43034b71abcccd96d85b3f42, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:31.622725 5313 raft_consensus.cc:2781] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:31.622859 5313 raft_consensus.cc:482] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:31.622997 5313 raft_consensus.cc:3037] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.625056 5313 raft_consensus.cc:504] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.625780 5313 leader_election.cc:288] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 election: Requested vote from peers 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:31.626377 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cc0ed38c34394a96e8818c39c8709" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f"
I20260927 15:19:31.626452 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cc0ed38c34394a96e8818c39c8709" candidate_uuid: "b5aa60e529054dabb988486b59675e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6de3cf43034b71abcccd96d85b3f42"
I20260927 15:19:31.626612 5238 raft_consensus.cc:3037] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.626811 4963 raft_consensus.cc:3037] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.628726 5238 raft_consensus.cc:2454] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 1.
I20260927 15:19:31.629218 5031 leader_election.cc:302] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:31.629513 5313 raft_consensus.cc:2781] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:31.629735 5313 raft_consensus.cc:686] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [term 1 LEADER]: Becoming Leader. State: Replica: b5aa60e529054dabb988486b59675e75, State: Running, Role: LEADER
I20260927 15:19:31.629709 4963 raft_consensus.cc:2454] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5aa60e529054dabb988486b59675e75 in term 1.
I20260927 15:19:31.630055 5313 consensus_queue.cc:237] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.633289 5318 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: TxnStatusTablet state changed. Reason: New leader b5aa60e529054dabb988486b59675e75. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.633693 5318 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:31.634572 5330 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:19:31.634352 4801 catalog_manager.cc:5495] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5aa60e529054dabb988486b59675e75 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:31.634999 5330 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:19:31.664565 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.658370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.2.246.66:44299)
0927 15:19:31.658633 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.658640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.658665 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:19:31.658982 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.658987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.659083 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.659259 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:31.659652 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.660504 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.661900 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.662382 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.662534 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.663110 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:31.663838 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:31.664036 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.664121 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.664215 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:19:31.664816 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.658208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44299 (local address 127.0.0.1:56654)
0927 15:19:31.658489 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.658514 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.658716 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.659281 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.659285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.659297 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:31.659520 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.659529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.660633 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.660639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.661686 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.661698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.662570 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.662577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.662796 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.662866 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:31.663990 (+ 1124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:31.664014 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.664103 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.664377 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:19:31.676612 4963 raft_consensus.cc:1261] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Refusing update from remote peer b5aa60e529054dabb988486b59675e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:31.676930 5238 raft_consensus.cc:1261] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Refusing update from remote peer b5aa60e529054dabb988486b59675e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:31.677825 5318 consensus_queue.cc:1035] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:31.678272 5313 consensus_queue.cc:1035] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:31.692597 5315 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: TxnStatusTablet state changed. Reason: New leader b5aa60e529054dabb988486b59675e75. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.692955 5315 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.692888 5314 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: TxnStatusTablet state changed. Reason: New leader b5aa60e529054dabb988486b59675e75. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.693215 5314 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.707952 5314 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.708307 5314 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 7c6de3cf43034b71abcccd96d85b3f42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.709720 5318 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.710098 5318 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:31.712338 5338 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.714375 5338 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P b5aa60e529054dabb988486b59675e75: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:19:31.714722 5335 mvcc.cc:205] Tried to move back new op lower bound from 7333979634372870144 to 7333979634199912448. Current Snapshot: MvccSnapshot[applied={T|T < 7333979634372870144}]
I20260927 15:19:31.716149 5315 tablet_replica.cc:439] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5aa60e529054dabb988486b59675e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } } }
I20260927 15:19:31.716432 5315 tablet_replica.cc:442] T 3f6cc0ed38c34394a96e8818c39c8709 P 72f3a19244ff4a80b6058ef23617451f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:19:31.729138 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.723344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.2.246.66:44299)
0927 15:19:31.723577 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.723584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.723606 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:31.723703 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.723709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.723793 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.723901 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:31.724303 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.725275 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.727127 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.727654 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.727816 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.728285 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:31.728428 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:31.728540 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.728693 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.728780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:19:31.750780 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.744077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40837 (local address 127.2.246.66:44299)
0927 15:19:31.744290 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.744297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.745137 (+ 840us) server_negotiation.cc:407] Connection header received
0927 15:19:31.745386 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.745393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.745460 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.745554 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:31.746263 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.747045 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.748775 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.749233 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.749377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.749926 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:31.750036 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:31.750134 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.750294 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.750406 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":55}
I20260927 15:19:31.751081 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.744048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44299 (local address 127.2.246.66:40837)
0927 15:19:31.744980 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.744998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.745244 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.745573 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.745577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.745587 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:31.746076 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.746089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.747224 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.747229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.748609 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.748627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.749454 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.749461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.749596 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.749660 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:31.750144 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:31.750160 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.750267 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.750454 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":1,"thread_start_us":492,"threads_started":1}
I20260927 15:19:31.763489 4963 raft_consensus.cc:1261] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Refusing update from remote peer b5aa60e529054dabb988486b59675e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:31.764168 5238 raft_consensus.cc:1261] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Refusing update from remote peer b5aa60e529054dabb988486b59675e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:31.764122 5338 consensus_queue.cc:1035] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:31.765139 5338 consensus_queue.cc:1035] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:31.770478 5333 mvcc.cc:205] Tried to move back new op lower bound from 7333979634737954816 to 7333979633547468800. Current Snapshot: MvccSnapshot[applied={T|T < 7333979634737954816}]
I20260927 15:19:31.848960 4800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:31.849247 4800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:31.854630 4800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Sending DeleteTablet for 3 replicas of tablet f6cb7ac5d24e473986702f8abc2de5d1
I20260927 15:19:31.856323 4938 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:31 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:38294
I20260927 15:19:31.856285 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:31 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:56648
I20260927 15:19:31.857450 5350 tablet_replica.cc:330] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: stopping tablet replica
I20260927 15:19:31.857880 5218 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb7ac5d24e473986702f8abc2de5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:31 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:41330
I20260927 15:19:31.858410 5350 raft_consensus.cc:2229] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:31.860831 4800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:31.861745 4800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:31.869794 5350 raft_consensus.cc:2258] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:31.871764 5349 tablet_replica.cc:330] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: stopping tablet replica
I20260927 15:19:31.872439 5349 raft_consensus.cc:2229] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:31.872903 5349 raft_consensus.cc:2258] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:31.873154 5351 tablet_replica.cc:330] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: stopping tablet replica
I20260927 15:19:31.874694 5217 tablet_service.cc:1467] Processing CreateTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 (DEFAULT_TABLE table=test-table [id=120b5d393b8444af922139d3583653f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.874642 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 (DEFAULT_TABLE table=test-table [id=120b5d393b8444af922139d3583653f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.875226 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35df4062d69b4548b4226b7bd3a682b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.875314 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35df4062d69b4548b4226b7bd3a682b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.878590 5351 raft_consensus.cc:2229] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:31.879261 5351 raft_consensus.cc:2258] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:31.883291 5308 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Bootstrap starting.
I20260927 15:19:31.885010 5308 tablet_bootstrap.cc:654] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.885665 5351 ts_tablet_manager.cc:1905] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:31.874253 4938 tablet_service.cc:1467] Processing CreateTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 (DEFAULT_TABLE table=test-table [id=120b5d393b8444af922139d3583653f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:31.890525 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35df4062d69b4548b4226b7bd3a682b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.892874 5349 ts_tablet_manager.cc:1905] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:31.896374 5350 ts_tablet_manager.cc:1905] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:31.900115 5351 ts_tablet_manager.cc:1918] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:31.900468 5351 log.cc:1196] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/f6cb7ac5d24e473986702f8abc2de5d1
I20260927 15:19:31.902454 5307 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Bootstrap starting.
I20260927 15:19:31.903770 5351 ts_tablet_manager.cc:1939] T f6cb7ac5d24e473986702f8abc2de5d1 P 72f3a19244ff4a80b6058ef23617451f: Deleting consensus metadata
I20260927 15:19:31.904618 5307 tablet_bootstrap.cc:654] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.906306 5309 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Bootstrap starting.
I20260927 15:19:31.907387 4788 catalog_manager.cc:4841] TS 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629): tablet f6cb7ac5d24e473986702f8abc2de5d1 (table test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]) successfully deleted
I20260927 15:19:31.908656 5309 tablet_bootstrap.cc:654] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.913651 5308 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: No bootstrap required, opened a new log
I20260927 15:19:31.913965 5308 ts_tablet_manager.cc:1397] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.002s
I20260927 15:19:31.914981 5308 raft_consensus.cc:348] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.915288 5308 raft_consensus.cc:374] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.915383 5308 raft_consensus.cc:729] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a19244ff4a80b6058ef23617451f, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.915860 5308 consensus_queue.cc:260] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.918942 5309 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: No bootstrap required, opened a new log
I20260927 15:19:31.919157 5309 ts_tablet_manager.cc:1397] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260927 15:19:31.919598 5308 ts_tablet_manager.cc:1428] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:19:31.919992 5309 raft_consensus.cc:348] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.920321 5309 raft_consensus.cc:374] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.920440 5309 raft_consensus.cc:729] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.920771 5309 consensus_queue.cc:260] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.921475 5309 ts_tablet_manager.cc:1428] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:31.928696 5350 ts_tablet_manager.cc:1918] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:31.928946 5350 log.cc:1196] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/f6cb7ac5d24e473986702f8abc2de5d1
I20260927 15:19:31.929814 5350 ts_tablet_manager.cc:1939] T f6cb7ac5d24e473986702f8abc2de5d1 P b5aa60e529054dabb988486b59675e75: Deleting consensus metadata
I20260927 15:19:31.931422 5307 tablet_bootstrap.cc:492] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: No bootstrap required, opened a new log
I20260927 15:19:31.931674 5307 ts_tablet_manager.cc:1397] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.016s
I20260927 15:19:31.932004 4787 catalog_manager.cc:4841] TS b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299): tablet f6cb7ac5d24e473986702f8abc2de5d1 (table test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]) successfully deleted
I20260927 15:19:31.932653 5307 raft_consensus.cc:348] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.932948 5307 raft_consensus.cc:374] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.933061 5307 raft_consensus.cc:729] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa60e529054dabb988486b59675e75, State: Initialized, Role: FOLLOWER
I20260927 15:19:31.933399 5307 consensus_queue.cc:260] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:31.934140 5349 ts_tablet_manager.cc:1918] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:31.934374 5349 log.cc:1196] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/f6cb7ac5d24e473986702f8abc2de5d1
I20260927 15:19:31.935078 5349 ts_tablet_manager.cc:1939] T f6cb7ac5d24e473986702f8abc2de5d1 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting consensus metadata
I20260927 15:19:31.937402 4787 catalog_manager.cc:4841] TS 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555): tablet f6cb7ac5d24e473986702f8abc2de5d1 (table test-table [id=7a9a036dc0c740f7b2e41e76e8871ca7]) successfully deleted
I20260927 15:19:31.937813 5307 ts_tablet_manager.cc:1428] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:19:32.198002 5314 raft_consensus.cc:482] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:32.198257 5314 raft_consensus.cc:504] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:32.199826 5314 leader_election.cc:288] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:32.211248 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.200245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:35565 (local address 127.2.246.66:44299)
0927 15:19:32.200489 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.200500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.201205 (+ 705us) server_negotiation.cc:407] Connection header received
0927 15:19:32.201559 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.201568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.201670 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.201796 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.202788 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.204291 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.207580 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.209525 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.209705 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.209821 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.210577 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.210792 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:19:32.212774 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.200045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44299 (local address 127.2.246.65:35565)
0927 15:19:32.201063 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.201087 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.201346 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.202097 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.202105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.202141 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.202517 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.202534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.204541 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.204549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.207353 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.207373 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.210067 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.210076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.210327 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.210399 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.212047 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.212264 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"threads_started":1}
I20260927 15:19:32.214841 5098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35df4062d69b4548b4226b7bd3a682b8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75" is_pre_election: true
I20260927 15:19:32.215435 5098 raft_consensus.cc:2454] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:32.216518 4890 leader_election.cc:302] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6de3cf43034b71abcccd96d85b3f42, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:32.217463 5314 raft_consensus.cc:2781] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:32.217638 5314 raft_consensus.cc:482] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:32.217747 5314 raft_consensus.cc:3037] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.221254 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.200362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:41629 (local address 127.2.246.65:45241)
0927 15:19:32.213239 (+ 12877us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.213266 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.213414 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.214285 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.214292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.214324 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.214440 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.214447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.215994 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.216002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.218698 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.218725 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.220176 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.220183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.220487 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.220523 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.220681 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.220869 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12735,"thread_start_us":1463,"threads_started":1}
I20260927 15:19:32.221143 5314 raft_consensus.cc:504] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:32.221686 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.201568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:45241 (local address 127.2.246.67:41629)
0927 15:19:32.213765 (+ 12197us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.213773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.213817 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:19:32.213957 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.213964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.214066 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.214250 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.214620 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.215744 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.218965 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.219919 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.220195 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.220294 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.220715 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.220880 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11977,"thread_start_us":12274,"threads_started":1}
I20260927 15:19:32.222266 5314 leader_election.cc:288] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Requested vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:32.222544 5237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35df4062d69b4548b4226b7bd3a682b8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f" is_pre_election: true
I20260927 15:19:32.222746 5098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35df4062d69b4548b4226b7bd3a682b8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75"
I20260927 15:19:32.222927 5237 raft_consensus.cc:2454] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:32.222995 5098 raft_consensus.cc:3037] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.224313 5237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35df4062d69b4548b4226b7bd3a682b8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f"
I20260927 15:19:32.224577 5237 raft_consensus.cc:3037] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.226794 5098 raft_consensus.cc:2454] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:32.227324 4890 leader_election.cc:302] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6de3cf43034b71abcccd96d85b3f42, b5aa60e529054dabb988486b59675e75; no voters:
I20260927 15:19:32.227687 5314 raft_consensus.cc:2781] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:32.228312 5314 raft_consensus.cc:686] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Running, Role: LEADER
I20260927 15:19:32.228307 5237 raft_consensus.cc:2454] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:32.228842 5314 consensus_queue.cc:237] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:32.233526 4801 catalog_manager.cc:5495] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "7c6de3cf43034b71abcccd96d85b3f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:32.275969 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.267062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.2.246.65:33555)
0927 15:19:32.267815 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.267821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.267849 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:19:32.267965 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.267971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.268084 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.268302 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:32.268881 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.270202 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.272038 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.272726 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.272865 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.273409 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.274273 (+ 864us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.275064 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.275161 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.275339 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":420,"threads_started":1}
I20260927 15:19:32.298238 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.291163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:34869 (local address 127.2.246.66:44299)
0927 15:19:32.291412 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.291418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.291928 (+ 510us) server_negotiation.cc:407] Connection header received
0927 15:19:32.292152 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.292156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.292244 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.292345 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:32.292991 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.293903 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.296036 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.296546 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.296680 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.297282 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.297471 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.297567 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.297732 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.297821 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:19:32.298358 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.290940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44299 (local address 127.2.246.65:34869)
0927 15:19:32.291760 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.291783 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.291979 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.292394 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.292399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.292414 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:32.292805 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.292817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.294063 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.294069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.295840 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.295855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.296772 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.296780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.296967 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.297051 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.297590 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.297602 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.297701 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.297899 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":356,"threads_started":1}
I20260927 15:19:32.310408 5237 raft_consensus.cc:1261] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:32.310843 5098 raft_consensus.cc:1261] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:32.311259 5358 consensus_queue.cc:1035] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:32.311805 5314 consensus_queue.cc:1035] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:32.322824 5334 mvcc.cc:205] Tried to move back new op lower bound from 7333979636976095232 to 7333979636654694400. Current Snapshot: MvccSnapshot[applied={T|T < 7333979636976095232}]
I20260927 15:19:32.356155 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.347390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:50769 (local address 127.2.246.65:33555)
0927 15:19:32.347678 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.347684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.348521 (+ 837us) server_negotiation.cc:407] Connection header received
0927 15:19:32.348785 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.348791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.348888 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.348991 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:32.349689 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.350785 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.353574 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.354386 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.354598 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.355178 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.355362 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.355466 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.355682 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.355792 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 15:19:32.356242 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.347590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.2.246.66:50769)
0927 15:19:32.348330 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.348402 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.348614 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.349182 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.349199 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.349214 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:19:32.349525 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.349537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.350967 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.350973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.353332 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.353347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.354765 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.354771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.354923 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.354971 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.355502 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.355519 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.355630 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.355809 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":398,"threads_started":1}
W20260927 15:19:32.357977 5334 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:19:33.982435 4799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:33.982752 4799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:33.988943 4799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Sending DeleteTablet for 3 replicas of tablet 35df4062d69b4548b4226b7bd3a682b8
I20260927 15:19:33.990111 4938 tablet_service.cc:1514] Processing DeleteTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:33 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:38294
I20260927 15:19:33.990185 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:33 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:56648
I20260927 15:19:33.990212 5217 tablet_service.cc:1514] Processing DeleteTablet for tablet 35df4062d69b4548b4226b7bd3a682b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:33 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:41330
I20260927 15:19:34.002568 5397 tablet_replica.cc:330] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: stopping tablet replica
I20260927 15:19:34.002652 5398 tablet_replica.cc:330] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: stopping tablet replica
I20260927 15:19:34.002910 5397 raft_consensus.cc:2229] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:34.002910 5398 raft_consensus.cc:2229] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:34.003181 5398 raft_consensus.cc:2258] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.003633 5397 raft_consensus.cc:2258] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.004660 5398 ts_tablet_manager.cc:1905] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:34.004729 5399 tablet_replica.cc:330] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: stopping tablet replica
I20260927 15:19:34.004915 5399 raft_consensus.cc:2229] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:34.005157 5399 raft_consensus.cc:2258] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.005429 5397 ts_tablet_manager.cc:1905] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:34.006152 5399 ts_tablet_manager.cc:1905] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:34.012720 5397 ts_tablet_manager.cc:1918] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:34.012943 5397 log.cc:1196] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/35df4062d69b4548b4226b7bd3a682b8
I20260927 15:19:34.013386 5399 ts_tablet_manager.cc:1918] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:34.013540 5397 ts_tablet_manager.cc:1939] T 35df4062d69b4548b4226b7bd3a682b8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting consensus metadata
I20260927 15:19:34.013622 5399 log.cc:1196] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/35df4062d69b4548b4226b7bd3a682b8
I20260927 15:19:34.013629 5398 ts_tablet_manager.cc:1918] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:34.013947 5398 log.cc:1196] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/35df4062d69b4548b4226b7bd3a682b8
I20260927 15:19:34.014479 5399 ts_tablet_manager.cc:1939] T 35df4062d69b4548b4226b7bd3a682b8 P b5aa60e529054dabb988486b59675e75: Deleting consensus metadata
I20260927 15:19:34.014499 5398 ts_tablet_manager.cc:1939] T 35df4062d69b4548b4226b7bd3a682b8 P 72f3a19244ff4a80b6058ef23617451f: Deleting consensus metadata
I20260927 15:19:34.014895 4787 catalog_manager.cc:4841] TS 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555): tablet 35df4062d69b4548b4226b7bd3a682b8 (table test-table [id=120b5d393b8444af922139d3583653f9]) successfully deleted
I20260927 15:19:34.015561 4787 catalog_manager.cc:4841] TS b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299): tablet 35df4062d69b4548b4226b7bd3a682b8 (table test-table [id=120b5d393b8444af922139d3583653f9]) successfully deleted
I20260927 15:19:34.015867 4788 catalog_manager.cc:4841] TS 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629): tablet 35df4062d69b4548b4226b7bd3a682b8 (table test-table [id=120b5d393b8444af922139d3583653f9]) successfully deleted
Sep 27 15:19:34 dist-test-slave-14jq krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:19:34 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:34.068301 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:34.046044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.2.246.126:44617)
0927 15:19:34.046735 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:34.046741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:34.046773 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:19:34.046971 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:34.046976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:34.047044 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:34.047153 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:34.048877 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:34.049708 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:34.051293 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:34.051819 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:34.051926 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:34.061997 (+ 10071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:34.062003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:34.062007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:34.062053 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:34.065175 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:34.066315 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:34.066319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:34.066322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:34.066467 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:34.066857 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:34.066860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:34.066863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:34.067126 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:34.067362 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:34.067666 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 15:19:34.067862 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
I20260927 15:19:34.072553 4799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:34.073201 4799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:34.081880 4938 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 (DEFAULT_TABLE table=test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.082042 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 (DEFAULT_TABLE table=test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.082398 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7880bf3ab4dabb6039654f9a277c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.082422 5217 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 (DEFAULT_TABLE table=test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.082598 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7880bf3ab4dabb6039654f9a277c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.083285 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7880bf3ab4dabb6039654f9a277c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.094568 5409 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Bootstrap starting.
I20260927 15:19:34.095054 5408 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Bootstrap starting.
I20260927 15:19:34.097402 5408 tablet_bootstrap.cc:654] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.097402 5409 tablet_bootstrap.cc:654] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.098335 5410 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Bootstrap starting.
I20260927 15:19:34.100952 5410 tablet_bootstrap.cc:654] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.103061 5409 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: No bootstrap required, opened a new log
I20260927 15:19:34.103327 5409 ts_tablet_manager.cc:1397] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:19:34.104364 5409 raft_consensus.cc:348] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.104668 5409 raft_consensus.cc:374] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.104766 5409 raft_consensus.cc:729] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.105080 5409 consensus_queue.cc:260] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.106575 5408 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: No bootstrap required, opened a new log
I20260927 15:19:34.106777 5408 ts_tablet_manager.cc:1397] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 15:19:34.107036 5409 ts_tablet_manager.cc:1428] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:34.108147 5408 raft_consensus.cc:348] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.108469 5408 raft_consensus.cc:374] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.108611 5408 raft_consensus.cc:729] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa60e529054dabb988486b59675e75, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.108976 5410 tablet_bootstrap.cc:492] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: No bootstrap required, opened a new log
I20260927 15:19:34.108950 5408 consensus_queue.cc:260] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.109198 5410 ts_tablet_manager.cc:1397] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260927 15:19:34.110062 5410 raft_consensus.cc:348] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.110443 5410 raft_consensus.cc:374] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.110559 5410 raft_consensus.cc:729] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a19244ff4a80b6058ef23617451f, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.110790 5410 consensus_queue.cc:260] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.113003 5408 ts_tablet_manager.cc:1428] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:19:34.113193 5410 ts_tablet_manager.cc:1428] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 15:19:34.211875 5388 raft_consensus.cc:482] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:34.212078 5388 raft_consensus.cc:504] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.212894 5388 leader_election.cc:288] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:34.213616 5097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7880bf3ab4dabb6039654f9a277c8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75" is_pre_election: true
I20260927 15:19:34.213660 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7880bf3ab4dabb6039654f9a277c8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f" is_pre_election: true
I20260927 15:19:34.213924 5238 raft_consensus.cc:2454] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:34.213980 5097 raft_consensus.cc:2454] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:34.214460 4891 leader_election.cc:302] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, 7c6de3cf43034b71abcccd96d85b3f42; no voters:
I20260927 15:19:34.214792 5388 raft_consensus.cc:2781] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:34.214931 5388 raft_consensus.cc:482] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:34.215036 5388 raft_consensus.cc:3037] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.216974 5388 raft_consensus.cc:504] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.217589 5388 leader_election.cc:288] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Requested vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:34.218289 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7880bf3ab4dabb6039654f9a277c8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f"
I20260927 15:19:34.218492 5238 raft_consensus.cc:3037] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.218379 5097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7880bf3ab4dabb6039654f9a277c8" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75"
I20260927 15:19:34.218688 5097 raft_consensus.cc:3037] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.220566 5238 raft_consensus.cc:2454] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:34.221148 4891 leader_election.cc:302] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, 7c6de3cf43034b71abcccd96d85b3f42; no voters:
I20260927 15:19:34.221235 5097 raft_consensus.cc:2454] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:34.221653 5388 raft_consensus.cc:2781] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:34.221951 5388 raft_consensus.cc:686] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Running, Role: LEADER
I20260927 15:19:34.222278 5388 consensus_queue.cc:237] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.226063 4799 catalog_manager.cc:5495] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "7c6de3cf43034b71abcccd96d85b3f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:34.278153 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:34.271270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.2.246.65:33555)
0927 15:19:34.271975 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:34.271979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:34.272003 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:19:34.272067 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:34.272071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:34.272138 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:34.272235 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:34.272668 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:34.273442 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:34.275180 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:34.276027 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:34.276193 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:34.276986 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:34.277174 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:34.277286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:34.277508 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:19:34.277607 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":416,"threads_started":1}
I20260927 15:19:34.291846 5238 raft_consensus.cc:1261] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:34.292011 5097 raft_consensus.cc:1261] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:34.292515 5388 consensus_queue.cc:1035] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:34.292929 5314 consensus_queue.cc:1035] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:34.348050 4799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:34.348400 4799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:34.352306 4799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Sending DeleteTablet for 3 replicas of tablet 2af7880bf3ab4dabb6039654f9a277c8
I20260927 15:19:34.353384 4938 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:34 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:38294
I20260927 15:19:34.353504 5217 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:34 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:41330
I20260927 15:19:34.353900 5397 tablet_replica.cc:330] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: stopping tablet replica
I20260927 15:19:34.354161 5397 raft_consensus.cc:2229] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:34.354267 5398 tablet_replica.cc:330] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: stopping tablet replica
I20260927 15:19:34.354489 5398 raft_consensus.cc:2229] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:34.354588 5397 raft_consensus.cc:2258] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.354699 5398 raft_consensus.cc:2258] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.355064 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7880bf3ab4dabb6039654f9a277c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:34 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:56648
I20260927 15:19:34.355392 5399 tablet_replica.cc:330] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: stopping tablet replica
I20260927 15:19:34.355621 5399 raft_consensus.cc:2229] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:34.355830 5399 raft_consensus.cc:2258] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:34.355847 5397 ts_tablet_manager.cc:1905] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:34.355118 4799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:19:34.356336 5398 ts_tablet_manager.cc:1905] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:19:34.356654 4799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:34.363102 5397 ts_tablet_manager.cc:1918] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:34.363337 5397 log.cc:1196] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/2af7880bf3ab4dabb6039654f9a277c8
I20260927 15:19:34.363848 5397 ts_tablet_manager.cc:1939] T 2af7880bf3ab4dabb6039654f9a277c8 P 7c6de3cf43034b71abcccd96d85b3f42: Deleting consensus metadata
I20260927 15:19:34.363950 5398 ts_tablet_manager.cc:1918] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:34.364284 5398 log.cc:1196] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/2af7880bf3ab4dabb6039654f9a277c8
I20260927 15:19:34.364873 5398 ts_tablet_manager.cc:1939] T 2af7880bf3ab4dabb6039654f9a277c8 P 72f3a19244ff4a80b6058ef23617451f: Deleting consensus metadata
I20260927 15:19:34.365249 4787 catalog_manager.cc:4841] TS 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555): tablet 2af7880bf3ab4dabb6039654f9a277c8 (table test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]) successfully deleted
I20260927 15:19:34.365904 5399 ts_tablet_manager.cc:1905] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:34.367376 5217 tablet_service.cc:1467] Processing CreateTablet for tablet 991198062109425d96d82afde348b4fd (DEFAULT_TABLE table=test-table [id=8f5b98f12cae48db86e980bd19df59b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.367614 4788 catalog_manager.cc:4841] TS 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629): tablet 2af7880bf3ab4dabb6039654f9a277c8 (table test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]) successfully deleted
I20260927 15:19:34.367862 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991198062109425d96d82afde348b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.367810 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 991198062109425d96d82afde348b4fd (DEFAULT_TABLE table=test-table [id=8f5b98f12cae48db86e980bd19df59b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.368338 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991198062109425d96d82afde348b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.368490 4938 tablet_service.cc:1467] Processing CreateTablet for tablet 991198062109425d96d82afde348b4fd (DEFAULT_TABLE table=test-table [id=8f5b98f12cae48db86e980bd19df59b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:34.368999 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991198062109425d96d82afde348b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:34.374768 5399 ts_tablet_manager.cc:1918] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:19:34.374984 5399 log.cc:1196] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/2af7880bf3ab4dabb6039654f9a277c8
I20260927 15:19:34.375438 5410 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Bootstrap starting.
I20260927 15:19:34.375559 5399 ts_tablet_manager.cc:1939] T 2af7880bf3ab4dabb6039654f9a277c8 P b5aa60e529054dabb988486b59675e75: Deleting consensus metadata
I20260927 15:19:34.376926 4787 catalog_manager.cc:4841] TS b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299): tablet 2af7880bf3ab4dabb6039654f9a277c8 (table test-table [id=5824e92ae70a4e339a3b38e9cbe7ad07]) successfully deleted
I20260927 15:19:34.377408 5410 tablet_bootstrap.cc:654] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.377574 5409 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Bootstrap starting.
I20260927 15:19:34.378006 5408 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Bootstrap starting.
I20260927 15:19:34.379436 5409 tablet_bootstrap.cc:654] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.380053 5410 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: No bootstrap required, opened a new log
I20260927 15:19:34.380055 5408 tablet_bootstrap.cc:654] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:34.380219 5410 ts_tablet_manager.cc:1397] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 15:19:34.381304 5410 raft_consensus.cc:348] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.381688 5410 raft_consensus.cc:374] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.381778 5410 raft_consensus.cc:729] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a19244ff4a80b6058ef23617451f, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.382529 5408 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: No bootstrap required, opened a new log
I20260927 15:19:34.382841 5408 ts_tablet_manager.cc:1397] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:19:34.382671 5410 consensus_queue.cc:260] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.383368 5410 ts_tablet_manager.cc:1428] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:19:34.384631 5408 raft_consensus.cc:348] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.384934 5408 raft_consensus.cc:374] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.385037 5408 raft_consensus.cc:729] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aa60e529054dabb988486b59675e75, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.385100 5409 tablet_bootstrap.cc:492] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: No bootstrap required, opened a new log
I20260927 15:19:34.385367 5409 ts_tablet_manager.cc:1397] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:19:34.385344 5408 consensus_queue.cc:260] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.385993 5408 ts_tablet_manager.cc:1428] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:34.387283 5409 raft_consensus.cc:348] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.387562 5409 raft_consensus.cc:374] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:34.387633 5409 raft_consensus.cc:729] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Initialized, Role: FOLLOWER
I20260927 15:19:34.387912 5409 consensus_queue.cc:260] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.388470 5409 ts_tablet_manager.cc:1428] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260927 15:19:34.395533 5392 raft_consensus.cc:482] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:34.395766 5392 raft_consensus.cc:504] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.396574 5392 leader_election.cc:288] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:34.397200 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991198062109425d96d82afde348b4fd" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f" is_pre_election: true
I20260927 15:19:34.397249 5097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991198062109425d96d82afde348b4fd" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75" is_pre_election: true
I20260927 15:19:34.397496 5238 raft_consensus.cc:2454] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:34.397523 5097 raft_consensus.cc:2454] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 0.
I20260927 15:19:34.398002 4891 leader_election.cc:302] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, 7c6de3cf43034b71abcccd96d85b3f42; no voters:
I20260927 15:19:34.398380 5392 raft_consensus.cc:2781] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:34.398521 5392 raft_consensus.cc:482] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:34.398614 5392 raft_consensus.cc:3037] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.401151 5392 raft_consensus.cc:504] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.401937 5392 leader_election.cc:288] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Requested vote from peers b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299), 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629)
I20260927 15:19:34.402477 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991198062109425d96d82afde348b4fd" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a19244ff4a80b6058ef23617451f"
I20260927 15:19:34.402490 5097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991198062109425d96d82afde348b4fd" candidate_uuid: "7c6de3cf43034b71abcccd96d85b3f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5aa60e529054dabb988486b59675e75"
I20260927 15:19:34.402736 5238 raft_consensus.cc:3037] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.402757 5097 raft_consensus.cc:3037] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:34.405025 5238 raft_consensus.cc:2454] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:34.405161 5097 raft_consensus.cc:2454] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6de3cf43034b71abcccd96d85b3f42 in term 1.
I20260927 15:19:34.405519 4891 leader_election.cc:302] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f3a19244ff4a80b6058ef23617451f, 7c6de3cf43034b71abcccd96d85b3f42; no voters:
I20260927 15:19:34.405932 5392 raft_consensus.cc:2781] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:34.406100 5392 raft_consensus.cc:686] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6de3cf43034b71abcccd96d85b3f42, State: Running, Role: LEADER
I20260927 15:19:34.406409 5392 consensus_queue.cc:237] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } }
I20260927 15:19:34.409478 4800 catalog_manager.cc:5495] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "7c6de3cf43034b71abcccd96d85b3f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6de3cf43034b71abcccd96d85b3f42" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:34.441589 5238 raft_consensus.cc:1261] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:34.441708 5097 raft_consensus.cc:1261] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Refusing update from remote peer 7c6de3cf43034b71abcccd96d85b3f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:34.442425 5314 consensus_queue.cc:1035] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5aa60e529054dabb988486b59675e75" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:34.442790 5392 consensus_queue.cc:1035] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a19244ff4a80b6058ef23617451f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:19:34.463877 5380 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:19:36.056980 4800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:36.057304 4800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:36.062474 4800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 233cc872a7264bb9b9b2975495f5abc5: Sending DeleteTablet for 3 replicas of tablet 991198062109425d96d82afde348b4fd
I20260927 15:19:36.063675 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet 991198062109425d96d82afde348b4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:36 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:56648
I20260927 15:19:36.063599 4938 tablet_service.cc:1514] Processing DeleteTablet for tablet 991198062109425d96d82afde348b4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:36 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:38294
I20260927 15:19:36.063648 5217 tablet_service.cc:1514] Processing DeleteTablet for tablet 991198062109425d96d82afde348b4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:36 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:41330
I20260927 15:19:36.064581 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4872
I20260927 15:19:36.091187 5448 tablet_replica.cc:330] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: stopping tablet replica
I20260927 15:19:36.091874 5448 raft_consensus.cc:2229] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:36.093070 5448 raft_consensus.cc:2258] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:36.107473 5448 ts_tablet_manager.cc:1905] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:36.107326 5449 tablet_replica.cc:330] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: stopping tablet replica
I20260927 15:19:36.108469 5449 raft_consensus.cc:2229] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:36.109949 5450 tablet_replica.cc:330] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: stopping tablet replica
I20260927 15:19:36.110167 5450 raft_consensus.cc:2229] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:36.110342 5449 raft_consensus.cc:2258] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:36.110548 5450 raft_consensus.cc:2258] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:36.111965 5449 ts_tablet_manager.cc:1905] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:36.112031 5450 ts_tablet_manager.cc:1905] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:36.118775 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.067056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.2.246.65:33555)
0927 15:19:36.079411 (+ 12355us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:36.079433 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:36.079496 (+ 63us) server_negotiation.cc:407] Connection header received
0927 15:19:36.079612 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:36.079624 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:36.079776 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:36.084431 (+ 4655us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:36.084816 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.086533 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.090209 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.092523 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.100739 (+ 8216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:36.104840 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:36.104874 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:36.104912 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:36.105000 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:36.112696 (+ 7696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.113967 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.113979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.113985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.114295 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.115145 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.115151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.115155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.115704 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:36.117600 (+ 1896us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:36.117803 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 15:19:36.118216 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12079,"thread_start_us":12975,"threads_started":1}
W20260927 15:19:36.120216 5003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55662
I20260927 15:19:36.120653 5449 ts_tablet_manager.cc:1918] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:36.120886 5449 log.cc:1196] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/991198062109425d96d82afde348b4fd
W20260927 15:19:36.121225 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:36.121750 5449 ts_tablet_manager.cc:1939] T 991198062109425d96d82afde348b4fd P 7c6de3cf43034b71abcccd96d85b3f42: Deleting consensus metadata
I20260927 15:19:36.121820 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4872
I20260927 15:19:36.126983 5448 ts_tablet_manager.cc:1918] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:36.127167 5448 log.cc:1196] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/991198062109425d96d82afde348b4fd
I20260927 15:19:36.128007 5450 ts_tablet_manager.cc:1918] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:19:36.128188 5450 log.cc:1196] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/991198062109425d96d82afde348b4fd
I20260927 15:19:36.128208 5448 ts_tablet_manager.cc:1939] T 991198062109425d96d82afde348b4fd P b5aa60e529054dabb988486b59675e75: Deleting consensus metadata
I20260927 15:19:36.128652 5450 ts_tablet_manager.cc:1939] T 991198062109425d96d82afde348b4fd P 72f3a19244ff4a80b6058ef23617451f: Deleting consensus metadata
I20260927 15:19:36.129748 4787 catalog_manager.cc:4841] TS b5aa60e529054dabb988486b59675e75 (127.2.246.66:44299): tablet 991198062109425d96d82afde348b4fd (table test-table [id=8f5b98f12cae48db86e980bd19df59b6]) successfully deleted
I20260927 15:19:36.129992 4788 catalog_manager.cc:4841] TS 72f3a19244ff4a80b6058ef23617451f (127.2.246.67:41629): tablet 991198062109425d96d82afde348b4fd (table test-table [id=8f5b98f12cae48db86e980bd19df59b6]) successfully deleted
W20260927 15:19:36.149408 4787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:33555 (error 108)
I20260927 15:19:36.149717 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5012
I20260927 15:19:36.152441 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.151082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.0.0.1:55670)
0927 15:19:36.151792 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:36.152078 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
W20260927 15:19:36.152773 4787 catalog_manager.cc:4584] TS 7c6de3cf43034b71abcccd96d85b3f42 (127.2.246.65:33555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 991198062109425d96d82afde348b4fd: Network error: Client connection negotiation failed: client connection to 127.2.246.65:33555: connect: Connection refused (error 111)
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:36.171247 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.150678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.2.246.66:44299)
0927 15:19:36.151285 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:36.151293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:36.151316 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:19:36.151403 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:36.151424 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:36.151525 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:36.151674 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:36.153563 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.154614 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.156557 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.157027 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.157156 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:36.163198 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:36.163221 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:36.163245 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:36.163302 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:36.167463 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.168542 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.168550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.168552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.168688 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.169319 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.169324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.169328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.169781 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:36.170194 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:36.170724 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 15:19:36.170963 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":560,"threads_started":1}
W20260927 15:19:36.172178 5143 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53254
W20260927 15:19:36.173084 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:36.173354 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5012
I20260927 15:19:36.206851 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.206066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.0.0.1:55684)
0927 15:19:36.206415 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:36.206588 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 15:19:36.207114 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5152
I20260927 15:19:36.208287 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.207732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.0.0.1:55700)
0927 15:19:36.207966 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:36.208101 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:36.226373 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.208068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.2.246.67:41629)
0927 15:19:36.208694 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:36.208699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:36.208717 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:19:36.208770 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:36.208773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:36.208833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:36.208957 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:36.210445 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.211246 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.212606 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:36.213009 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:36.213147 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:36.218148 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:36.218170 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:36.218192 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:36.218252 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:36.222639 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.223725 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.223734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.223737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.223932 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:36.224354 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:36.224359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:36.224361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:36.224916 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:36.225249 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:36.225493 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:19:36.225734 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":316,"threads_started":1}
W20260927 15:19:36.227298 5276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188
W20260927 15:19:36.228107 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5152: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:36.228387 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5152
I20260927 15:19:36.247116 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 4769
I20260927 15:19:36.248708 4841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:36.389448 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:36.388811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33555 (local address 127.0.0.1:55704)
0927 15:19:36.389040 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:36.389163 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:19:36.389770 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 4769
2026-09-27T15:19:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9232 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[5460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[5460](info): set up 2 sockets
Sep 27 15:19:36 dist-test-slave-14jq krb5kdc[5460](info): commencing operation
krb5kdc: starting...
W20260927 15:19:37.490228 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:37 dist-test-slave-14jq krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:37.715013 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:41249
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:38.051157 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:38.051527 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:38.051674 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:38.059360 5473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:38.059520 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:38.059630 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:38.059727 5473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:38.059813 5473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:38.070159 5473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:38.071568 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:38.072849 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:38.087095 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:38.087129 5479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:38.087179 5478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:38.088038 5473 server_base.cc:1034] running on GCE node
I20260927 15:19:38.088961 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:38.090667 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:38.092010 5473 hybrid_clock.cc:648] HybridClock initialized: now 1790522378091926 us; error 73 us; skew 500 ppm
I20260927 15:19:38.092525 5473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:38 dist-test-slave-14jq krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:38.096264 5473 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:38.098538 5473 webserver.cc:460] Webserver started at http://127.2.246.126:39407/ using document root <none> and password file <none>
I20260927 15:19:38.099319 5473 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:38.099488 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:38.099829 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:38.103094 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "0f8cd3e5d6004d45b1313dfacf9bb727"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "3e69c8bb4b3cd7d62a651869b9779362"
server_key_iv: "a16ab50bbbe377968a8de8f5297c120b"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.103993 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "0f8cd3e5d6004d45b1313dfacf9bb727"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "3e69c8bb4b3cd7d62a651869b9779362"
server_key_iv: "a16ab50bbbe377968a8de8f5297c120b"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.109798 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:19:38.114455 5488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:38.116696 5473 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:19:38.116976 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "0f8cd3e5d6004d45b1313dfacf9bb727"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "3e69c8bb4b3cd7d62a651869b9779362"
server_key_iv: "a16ab50bbbe377968a8de8f5297c120b"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.117216 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:38.136543 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:38.142012 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:38.142379 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:38.165053 5473 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:41249
I20260927 15:19:38.165096 5539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:41249 every 8 connection(s)
I20260927 15:19:38.166658 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:38.170203 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5473
I20260927 15:19:38.170490 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:38.171320 3033 external_mini_cluster.cc:1428] Setting key 1443e2916116fdfc004f3243935db948
I20260927 15:19:38.172642 5540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:19:38 dist-test-slave-14jq krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:38.191128 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727: Bootstrap starting.
I20260927 15:19:38.195145 5540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:38.196527 5540 log.cc:824] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:38.198392 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:38.174118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.2.246.126:41249)
0927 15:19:38.175061 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:38.175074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:38.175122 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:19:38.176760 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:38.176783 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:38.177247 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:38.177639 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:38.180106 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.181615 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.183784 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.184516 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.184677 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:38.190150 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:38.190158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:38.190176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:38.190218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:38.193563 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:38.194420 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:38.194427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:38.194434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:38.194594 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:38.194975 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:38.194979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:38.194981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:38.195640 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:38.195969 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:38.196424 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 15:19:38.196876 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":427,"threads_started":1}
I20260927 15:19:38.201138 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727: No bootstrap required, opened a new log
I20260927 15:19:38.211272 5540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } }
I20260927 15:19:38.211864 5540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:38.211985 5540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8cd3e5d6004d45b1313dfacf9bb727, State: Initialized, Role: FOLLOWER
I20260927 15:19:38.212790 5540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } }
I20260927 15:19:38.213097 5540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:38.213259 5540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:38.213447 5540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:38.216028 5540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } }
I20260927 15:19:38.216691 5540 leader_election.cc:302] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f8cd3e5d6004d45b1313dfacf9bb727; no voters:
I20260927 15:19:38.217360 5540 leader_election.cc:288] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:38.217590 5545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:38.218612 5545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [term 1 LEADER]: Becoming Leader. State: Replica: 0f8cd3e5d6004d45b1313dfacf9bb727, State: Running, Role: LEADER
I20260927 15:19:38.219297 5545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } }
I20260927 15:19:38.220253 5540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:38.222344 5547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f8cd3e5d6004d45b1313dfacf9bb727. Latest consensus state: current_term: 1 leader_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } } }
I20260927 15:19:38.222816 5547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:38.222916 5546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8cd3e5d6004d45b1313dfacf9bb727" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41249 } } }
I20260927 15:19:38.223770 5546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:38.227679 5550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:38.233675 5550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:38.245774 5550 catalog_manager.cc:1347] Generated new cluster ID: 827ddbb48b854a67ad92a3546bec43cd
I20260927 15:19:38.245998 5550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:38.258666 5550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:38.260893 5550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:38.273149 5550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f8cd3e5d6004d45b1313dfacf9bb727: Generated new TSK 0
I20260927 15:19:38.274510 5550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:38.334372 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:38.731467 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:38.731755 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:38.731899 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:38.739547 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:38.739667 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:38.739847 5567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:38.749820 5567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:38.751235 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:38.752562 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:38.767417 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:38.767417 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:38.767398 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:38.769171 5567 server_base.cc:1034] running on GCE node
I20260927 15:19:38.770092 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:38.771391 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:38.772676 5567 hybrid_clock.cc:648] HybridClock initialized: now 1790522378772592 us; error 75 us; skew 500 ppm
I20260927 15:19:38.773135 5567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:38 dist-test-slave-14jq krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:38.778206 5567 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:38.780915 5567 webserver.cc:460] Webserver started at http://127.2.246.65:37713/ using document root <none> and password file <none>
I20260927 15:19:38.782033 5567 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:38.782229 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:38.782673 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:38.786773 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "e426801482334f31b18c08625d32d1ab"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "67389cc51ee1cc6597ac98c4e49158b1"
server_key_iv: "69d318d41c3eeb89b975545dc56e827f"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.787949 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "e426801482334f31b18c08625d32d1ab"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "67389cc51ee1cc6597ac98c4e49158b1"
server_key_iv: "69d318d41c3eeb89b975545dc56e827f"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.795373 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:19:38.800869 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:38.802965 5567 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:19:38.803241 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "e426801482334f31b18c08625d32d1ab"
format_stamp: "Formatted at 2026-09-27 15:19:38 on dist-test-slave-14jq"
server_key: "67389cc51ee1cc6597ac98c4e49158b1"
server_key_iv: "69d318d41c3eeb89b975545dc56e827f"
server_key_version: "encryptionkey@0"
I20260927 15:19:38.803495 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:38.824781 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:38.830504 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:38.830876 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:38.832132 5567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:38.833941 5567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:38.834020 5567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:38.834160 5567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:38.834230 5567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:38.874415 5567 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:43879
I20260927 15:19:38.874464 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:43879 every 8 connection(s)
I20260927 15:19:38.876127 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:38.881217 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5567
I20260927 15:19:38.881475 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:38.882270 3033 external_mini_cluster.cc:1428] Setting key 4d12b6ef34cbe64fbd86b2eecebb729b
Sep 27 15:19:38 dist-test-slave-14jq krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:38.901173 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:38.879393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:47511 (local address 127.2.246.126:41249)
0927 15:19:38.879916 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:38.879922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:38.881214 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 15:19:38.883698 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:38.883703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:38.883811 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:38.883934 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:38.885682 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.886451 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.888037 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.888434 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.888599 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:38.892977 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:38.892989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:38.892993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:38.893036 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:38.896784 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:38.898045 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:38.898054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:38.898060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:38.898248 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:38.898854 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:38.898861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:38.898866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:38.899279 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:38.899456 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:38.900448 (+ 992us) server_negotiation.cc:299] Negotiation successful
0927 15:19:38.900721 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260927 15:19:38.902311 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:38.880145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:41249 (local address 127.2.246.65:47511)
0927 15:19:38.881015 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:38.881071 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:38.883354 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:38.884232 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:38.884247 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:38.884886 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:38.885506 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:38.885524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.886606 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.886612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:38.887871 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:38.887883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:38.888682 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:38.888691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:38.888945 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:38.889798 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:38.889821 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:38.892647 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:38.897081 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:38.897093 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:38.897117 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:38.897818 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:38.898418 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:38.898423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:38.898434 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:38.898695 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:38.899432 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:38.899439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:38.899973 (+ 534us) client_negotiation.cc:769] Sending connection context
0927 15:19:38.900255 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 15:19:38.900778 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:19:38.904421 5695 heartbeater.cc:347] Connected to a master server at 127.2.246.126:41249
I20260927 15:19:38.904873 5695 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:38.905990 5695 heartbeater.cc:511] Master 127.2.246.126:41249 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:38.909588 5505 ts_manager.cc:194] Registered new tserver with Master: e426801482334f31b18c08625d32d1ab (127.2.246.65:43879)
I20260927 15:19:38.912256 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:47511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:38.934163 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:39.298096 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:39.298446 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:39.298547 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:39.306017 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:39.306142 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:39.306259 5701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:39.316453 5701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:39.317755 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:39.319118 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:39.335614 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:39.335594 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:39.335584 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:39.340204 5701 server_base.cc:1034] running on GCE node
I20260927 15:19:39.341003 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:39.342278 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:39.343737 5701 hybrid_clock.cc:648] HybridClock initialized: now 1790522379343643 us; error 86 us; skew 500 ppm
I20260927 15:19:39.344177 5701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:39 dist-test-slave-14jq krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:39.352381 5701 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:39.354435 5701 webserver.cc:460] Webserver started at http://127.2.246.66:45719/ using document root <none> and password file <none>
I20260927 15:19:39.355224 5701 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:39.355345 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:39.355721 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:39.358933 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "711ec5aac7024eb58be860c9f5f13491"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "fee7552d51ffb890b4b77f42c31c2f3d"
server_key_iv: "e929aadedcbbda17771a3cdb6db3a037"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.359817 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "711ec5aac7024eb58be860c9f5f13491"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "fee7552d51ffb890b4b77f42c31c2f3d"
server_key_iv: "e929aadedcbbda17771a3cdb6db3a037"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.366320 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 15:19:39.371078 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:39.373035 5701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:19:39.373306 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "711ec5aac7024eb58be860c9f5f13491"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "fee7552d51ffb890b4b77f42c31c2f3d"
server_key_iv: "e929aadedcbbda17771a3cdb6db3a037"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.373572 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:39.396374 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:39.403335 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:39.403793 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:39.405174 5701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:39.407212 5701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:39.407325 5701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:39.407486 5701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:39.407562 5701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:39.451592 5701 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:38309
I20260927 15:19:39.451668 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:38309 every 8 connection(s)
I20260927 15:19:39.453227 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:39.461020 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5701
I20260927 15:19:39.461266 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:39.461793 3033 external_mini_cluster.cc:1428] Setting key d4cd7f077bd592ba9e9d5568e9360517
Sep 27 15:19:39 dist-test-slave-14jq krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:39.475925 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:39.456349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:59409 (local address 127.2.246.126:41249)
0927 15:19:39.456846 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:39.456851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:39.458132 (+ 1281us) server_negotiation.cc:407] Connection header received
0927 15:19:39.460174 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:39.460180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:39.460329 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:39.460457 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:39.462297 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:39.463219 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:39.465009 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:39.465573 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:39.465710 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:39.469908 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:39.469915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:39.469920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:39.469966 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:39.472748 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:39.473575 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:39.473587 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:39.473589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:39.473721 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:39.474189 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:39.474201 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:39.474203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:39.474464 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:39.474610 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:39.475321 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 15:19:39.475536 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":273,"threads_started":1}
I20260927 15:19:39.477051 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:39.456926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:41249 (local address 127.2.246.66:59409)
0927 15:19:39.457896 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:39.457962 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:39.459883 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:39.460722 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:39.460740 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:39.461424 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:39.462118 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:39.462138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:39.463394 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:39.463398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:39.464853 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:39.464866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:39.465865 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:39.465874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:39.466135 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:39.466878 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:39.466902 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:39.469552 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:39.472938 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:39.472948 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:39.472968 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:39.473441 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:39.473855 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:39.473863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:39.473868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:39.474068 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:39.474587 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:39.474596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:39.474956 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 15:19:39.475226 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 15:19:39.475791 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":352,"threads_started":1}
I20260927 15:19:39.479044 5829 heartbeater.cc:347] Connected to a master server at 127.2.246.126:41249
I20260927 15:19:39.479393 5829 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:39.480140 5829 heartbeater.cc:511] Master 127.2.246.126:41249 requested a full tablet report, sending...
I20260927 15:19:39.481966 5505 ts_manager.cc:194] Registered new tserver with Master: 711ec5aac7024eb58be860c9f5f13491 (127.2.246.66:38309)
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:39.483702 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:59409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:39.514418 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:39.870442 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:39.870775 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:39.870939 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:39.878980 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:39.879146 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:39.879323 5835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:39.889367 5835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:39.890836 5835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:39.892166 5835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:39.906433 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:39.906540 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:39.906489 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:39.908187 5835 server_base.cc:1034] running on GCE node
I20260927 15:19:39.908986 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:39.910262 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:39.911556 5835 hybrid_clock.cc:648] HybridClock initialized: now 1790522379911489 us; error 65 us; skew 500 ppm
I20260927 15:19:39.911985 5835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:39 dist-test-slave-14jq krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:39.915990 5835 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:39.916970 5695 heartbeater.cc:503] Master 127.2.246.126:41249 was elected leader, sending a full tablet report...
I20260927 15:19:39.918069 5835 webserver.cc:460] Webserver started at http://127.2.246.67:38797/ using document root <none> and password file <none>
I20260927 15:19:39.919111 5835 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:39.919305 5835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:39.919731 5835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:39.923249 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "54661284aa8f406d806843de22743b53"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "bc9225c0112738501adcabb62ce5190d"
server_key_iv: "fe2b168d878fb58d26985dc8a686db22"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.924135 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "54661284aa8f406d806843de22743b53"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "bc9225c0112738501adcabb62ce5190d"
server_key_iv: "fe2b168d878fb58d26985dc8a686db22"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.930469 5835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:19:39.934942 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:39.936707 5835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:19:39.936946 5835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "54661284aa8f406d806843de22743b53"
format_stamp: "Formatted at 2026-09-27 15:19:39 on dist-test-slave-14jq"
server_key: "bc9225c0112738501adcabb62ce5190d"
server_key_iv: "fe2b168d878fb58d26985dc8a686db22"
server_key_version: "encryptionkey@0"
I20260927 15:19:39.937202 5835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:39.956347 5835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:39.961580 5835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:39.962013 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:39.963291 5835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:39.965031 5835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:39.965144 5835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:39.965315 5835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:39.965391 5835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:40.006726 5835 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:43421
I20260927 15:19:40.006783 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:43421 every 8 connection(s)
I20260927 15:19:40.008289 5835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:40.018437 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5835
I20260927 15:19:40.018692 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:40.019299 3033 external_mini_cluster.cc:1428] Setting key 96b80fea3b0d127a30f6819c06cf3327
Sep 27 15:19:40 dist-test-slave-14jq krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:40.035380 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:40.011851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:50039 (local address 127.2.246.126:41249)
0927 15:19:40.012540 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:40.012546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:40.013721 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 15:19:40.016047 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:40.016052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:40.016136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:40.016283 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:40.019195 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:40.020472 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:40.023297 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:40.023712 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:40.023822 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:40.028739 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:40.028744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:40.028748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:40.028788 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:40.031537 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:40.032639 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:40.032644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:40.032646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:40.032784 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:40.033274 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:40.033277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:40.033279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:40.033546 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:40.033666 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:40.034713 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0927 15:19:40.034957 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":378,"threads_started":1}
I20260927 15:19:40.036899 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:40.012504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:41249 (local address 127.2.246.67:50039)
0927 15:19:40.013469 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:40.013531 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:40.015629 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:40.016718 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:40.016730 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:40.017655 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:40.018869 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:40.018905 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:40.020699 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:40.020705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:40.023034 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:40.023065 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:40.024042 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:40.024050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:40.024365 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:40.025383 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:40.025414 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:40.028362 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:40.031801 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:40.031813 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:40.031839 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:40.032460 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:40.032904 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:40.032908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:40.032910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:40.033148 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:40.033714 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:40.033725 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:40.034283 (+ 558us) client_negotiation.cc:769] Sending connection context
0927 15:19:40.034637 (+ 354us) client_negotiation.cc:240] Negotiation successful
0927 15:19:40.035279 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":354,"threads_started":1}
I20260927 15:19:40.038914 5963 heartbeater.cc:347] Connected to a master server at 127.2.246.126:41249
I20260927 15:19:40.039500 5963 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:40.040493 5963 heartbeater.cc:511] Master 127.2.246.126:41249 requested a full tablet report, sending...
I20260927 15:19:40.042897 5505 ts_manager.cc:194] Registered new tserver with Master: 54661284aa8f406d806843de22743b53 (127.2.246.67:43421)
I20260927 15:19:40.043931 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:50039
I20260927 15:19:40.049789 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:19:40.090061 5965 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:19:40.086748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.2.246.126:41249)
0927 15:19:40.086962 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:40.086967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:40.087697 (+ 730us) server_negotiation.cc:407] Connection header received
0927 15:19:40.087883 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:40.087887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:40.087959 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:40.088082 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:40.089699 (+ 1617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:19:40.094290 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5567
W20260927 15:19:40.098110 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5567: Not authorized: Client connection negotiation failed: client connection to 127.2.246.65:43879: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:19:40.098279 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5567
I20260927 15:19:40.114591 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5701
W20260927 15:19:40.118578 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5701: Not authorized: Client connection negotiation failed: client connection to 127.2.246.66:38309: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:19:40.118860 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5701
I20260927 15:19:40.137467 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5835
W20260927 15:19:40.141359 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5835: Not authorized: Client connection negotiation failed: client connection to 127.2.246.67:43421: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:19:40.141606 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5835
I20260927 15:19:40.161039 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5473
I20260927 15:19:40.162812 5535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:40.287449 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5473
2026-09-27T15:19:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3895 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:40 dist-test-slave-14jq krb5kdc[5981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:40 dist-test-slave-14jq krb5kdc[5981](info): set up 2 sockets
Sep 27 15:19:40 dist-test-slave-14jq krb5kdc[5981](info): commencing operation
krb5kdc: starting...
W20260927 15:19:41.406352 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:41 dist-test-slave-14jq krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:41.646925 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:41247
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:41.977159 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:41.977516 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:41.977716 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:41.985280 5994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:41.985430 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:41.985535 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:41.985643 5994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:41.985721 5994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:41.996013 5994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:41.997449 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:41.998749 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:42.012585 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:42.012655 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:42.013397 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:42.014173 5994 server_base.cc:1034] running on GCE node
I20260927 15:19:42.015007 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:42.016497 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:42.017764 5994 hybrid_clock.cc:648] HybridClock initialized: now 1790522382017666 us; error 85 us; skew 500 ppm
I20260927 15:19:42.018168 5994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:42 dist-test-slave-14jq krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:42.021502 5994 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:42.023307 5994 webserver.cc:460] Webserver started at http://127.2.246.126:40097/ using document root <none> and password file <none>
I20260927 15:19:42.024010 5994 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:42.024158 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:42.024458 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:42.027508 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "9ea201a458b041b9b715b123076b27f6"
format_stamp: "Formatted at 2026-09-27 15:19:42 on dist-test-slave-14jq"
server_key: "38b5e1e830561117114986cbdd24ab1a"
server_key_iv: "e2c3872ada47fde1b21e845d4d404a31"
server_key_version: "encryptionkey@0"
I20260927 15:19:42.028293 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "9ea201a458b041b9b715b123076b27f6"
format_stamp: "Formatted at 2026-09-27 15:19:42 on dist-test-slave-14jq"
server_key: "38b5e1e830561117114986cbdd24ab1a"
server_key_iv: "e2c3872ada47fde1b21e845d4d404a31"
server_key_version: "encryptionkey@0"
I20260927 15:19:42.033910 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:19:42.037786 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:42.039669 5994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:19:42.039906 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "9ea201a458b041b9b715b123076b27f6"
format_stamp: "Formatted at 2026-09-27 15:19:42 on dist-test-slave-14jq"
server_key: "38b5e1e830561117114986cbdd24ab1a"
server_key_iv: "e2c3872ada47fde1b21e845d4d404a31"
server_key_version: "encryptionkey@0"
I20260927 15:19:42.040150 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:42.056900 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:42.061390 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:42.061681 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:42.084056 5994 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:41247
I20260927 15:19:42.084163 6060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:41247 every 8 connection(s)
I20260927 15:19:42.085677 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:42.089068 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 5994
I20260927 15:19:42.089310 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:42.090250 3033 external_mini_cluster.cc:1428] Setting key 129fcbc21a7c3b3d3b63ace1f70e8130
I20260927 15:19:42.091289 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:19:42 dist-test-slave-14jq krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:42.107373 6061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6: Bootstrap starting.
I20260927 15:19:42.111891 6061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:42.113754 6061 log.cc:824] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:42.114857 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:42.092740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.2.246.126:41247)
0927 15:19:42.093802 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:42.093817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:42.093942 (+ 125us) server_negotiation.cc:407] Connection header received
0927 15:19:42.095621 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:42.095646 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:42.096127 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:42.096588 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:42.098219 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:42.099651 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:42.100906 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:42.101697 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:42.101943 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:42.106456 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:42.106464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:42.106481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:42.106521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:42.109951 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:42.110652 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:42.110663 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:42.110676 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:42.110863 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:42.111239 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:42.111246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:42.111254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:42.111860 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:42.112191 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:42.112686 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 15:19:42.113208 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":527,"threads_started":1}
I20260927 15:19:42.118459 6061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6: No bootstrap required, opened a new log
I20260927 15:19:42.119573 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:19:42.127605 6061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } }
I20260927 15:19:42.128247 6061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:42.128348 6061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea201a458b041b9b715b123076b27f6, State: Initialized, Role: FOLLOWER
I20260927 15:19:42.129195 6061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } }
I20260927 15:19:42.129528 6061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:42.129706 6061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:42.129954 6061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:42.134362 6061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } }
I20260927 15:19:42.135252 6061 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea201a458b041b9b715b123076b27f6; no voters:
I20260927 15:19:42.136168 6061 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:42.136474 6067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:42.138110 6067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea201a458b041b9b715b123076b27f6, State: Running, Role: LEADER
I20260927 15:19:42.138844 6067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } }
I20260927 15:19:42.139753 6061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:42.141942 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea201a458b041b9b715b123076b27f6. Latest consensus state: current_term: 1 leader_uuid: "9ea201a458b041b9b715b123076b27f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } } }
I20260927 15:19:42.142575 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:42.143275 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ea201a458b041b9b715b123076b27f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea201a458b041b9b715b123076b27f6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 41247 } } }
I20260927 15:19:42.143615 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:42.145895 6072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:42.154007 6072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:42.176559 6072 catalog_manager.cc:1347] Generated new cluster ID: 2e0c2a8a411d448d874b407dc7cf069b
I20260927 15:19:42.176783 6072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:42.210819 6072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:42.213948 6072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:42.226750 6072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ea201a458b041b9b715b123076b27f6: Generated new TSK 0
I20260927 15:19:42.228096 6072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:19:42.231271 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:42.220162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.2.246.126:41247)
0927 15:19:42.220416 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:42.220424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:42.221462 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 15:19:42.221685 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:42.221691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:42.221797 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:42.221976 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:42.223501 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:42.224811 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:42.226789 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:42.227567 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:42.227742 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:42.229375 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:42.229384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:42.229389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:42.230344 (+ 955us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:42.230415 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:42.230453 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:42.230580 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:42.230754 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:19:42.230864 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:19:42.241073 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 5994
I20260927 15:19:42.242686 6056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:42.348884 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 5994
2026-09-27T15:19:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2057 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:42 dist-test-slave-14jq krb5kdc[6095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:42 dist-test-slave-14jq krb5kdc[6095](info): set up 2 sockets
Sep 27 15:19:42 dist-test-slave-14jq krb5kdc[6095](info): commencing operation
krb5kdc: starting...
W20260927 15:19:43.530368 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:43 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:43.762744 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:43615
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:44.094642 6108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:44.094964 6108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:44.095122 6108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:44.102510 6108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:44.102632 6108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:44.102679 6108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:44.102819 6108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:44.102938 6108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:44.112986 6108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:44.114503 6108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:44.115907 6108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:44.129473 6116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:44.129496 6113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:44.129549 6114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:44.131389 6108 server_base.cc:1034] running on GCE node
I20260927 15:19:44.132452 6108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:44.134186 6108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:44.135732 6108 hybrid_clock.cc:648] HybridClock initialized: now 1790522384135646 us; error 63 us; skew 500 ppm
I20260927 15:19:44.136220 6108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:44 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:44.140286 6108 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:44.142406 6108 webserver.cc:460] Webserver started at http://127.2.246.126:38995/ using document root <none> and password file <none>
I20260927 15:19:44.143181 6108 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:44.143342 6108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:44.143702 6108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:44.147007 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "14ce891e054f4a3f827714c1cdd359bb"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "6d81646a573b6e065d171259eb00d8cf"
server_key_iv: "bb05c9a7443ec318ce9565b657abf999"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.147869 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "14ce891e054f4a3f827714c1cdd359bb"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "6d81646a573b6e065d171259eb00d8cf"
server_key_iv: "bb05c9a7443ec318ce9565b657abf999"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.154066 6108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:19:44.158797 6123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:44.160848 6108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:44.161175 6108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "14ce891e054f4a3f827714c1cdd359bb"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "6d81646a573b6e065d171259eb00d8cf"
server_key_iv: "bb05c9a7443ec318ce9565b657abf999"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.161433 6108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:44.189646 6108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:44.194733 6108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:44.195013 6108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:44.219698 6108 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:43615
I20260927 15:19:44.219761 6174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:43615 every 8 connection(s)
I20260927 15:19:44.221418 6108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:44.226984 6175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:44.228006 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6108
I20260927 15:19:44.228259 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:44.228991 3033 external_mini_cluster.cc:1428] Setting key 47ab4e407d11442c773d3873c12af2e5
I20260927 15:19:44.244036 6175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb: Bootstrap starting.
Sep 27 15:19:44 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:44.249197 6175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:44.251050 6175 log.cc:824] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:44.256603 6175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb: No bootstrap required, opened a new log
I20260927 15:19:44.259186 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:44.231687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.2.246.126:43615)
0927 15:19:44.232850 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:44.232870 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:44.232920 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:19:44.235567 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:44.235602 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:44.236183 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:44.236684 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:44.238379 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:44.239796 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:44.241202 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:44.242119 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:44.242368 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:44.247270 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:44.247297 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:44.247327 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:44.247399 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:44.252521 (+ 5122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:44.253612 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:44.253627 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:44.253639 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:44.254003 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:44.254640 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:44.254650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:44.254658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:44.255638 (+ 980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:44.256062 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:44.256595 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 15:19:44.257216 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":467,"threads_started":1}
I20260927 15:19:44.269074 6175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } }
I20260927 15:19:44.269586 6175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:44.269673 6175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ce891e054f4a3f827714c1cdd359bb, State: Initialized, Role: FOLLOWER
I20260927 15:19:44.270476 6175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } }
I20260927 15:19:44.270747 6175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:44.270864 6175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:44.271051 6175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:44.273883 6175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } }
I20260927 15:19:44.274746 6175 leader_election.cc:302] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14ce891e054f4a3f827714c1cdd359bb; no voters:
I20260927 15:19:44.279829 6175 leader_election.cc:288] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:44.280253 6180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:44.289575 6175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:44.291146 6180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [term 1 LEADER]: Becoming Leader. State: Replica: 14ce891e054f4a3f827714c1cdd359bb, State: Running, Role: LEADER
I20260927 15:19:44.292451 6180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } }
I20260927 15:19:44.300058 6182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14ce891e054f4a3f827714c1cdd359bb. Latest consensus state: current_term: 1 leader_uuid: "14ce891e054f4a3f827714c1cdd359bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } } }
I20260927 15:19:44.300628 6182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:44.301316 6181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14ce891e054f4a3f827714c1cdd359bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ce891e054f4a3f827714c1cdd359bb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 43615 } } }
I20260927 15:19:44.301712 6181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:44.304659 6190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:44.310314 6190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:44.322844 6190 catalog_manager.cc:1347] Generated new cluster ID: f75e6b16220c44beb1873158bbe3809a
I20260927 15:19:44.323004 6190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:44.373734 6190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:44.375713 6190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:44.390606 6190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14ce891e054f4a3f827714c1cdd359bb: Generated new TSK 0
I20260927 15:19:44.391899 6190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:44.464174 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:44.862716 6201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:44.863030 6201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:44.863185 6201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:44.870841 6201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:44.870967 6201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:44.871155 6201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:44.881217 6201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:44.882691 6201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:44.884075 6201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:44.898900 6209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:44.898911 6206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:44.898967 6207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:44.899976 6201 server_base.cc:1034] running on GCE node
I20260927 15:19:44.900630 6201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:44.901597 6201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:44.902846 6201 hybrid_clock.cc:648] HybridClock initialized: now 1790522384902769 us; error 99 us; skew 500 ppm
I20260927 15:19:44.903231 6201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:44 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:44.909695 6201 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:44.911583 6201 webserver.cc:460] Webserver started at http://127.2.246.65:40351/ using document root <none> and password file <none>
I20260927 15:19:44.912319 6201 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:44.912426 6201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:44.912714 6201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:44.915849 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "84d6e93366d047178d0a8ad5fc9de195"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "356a2ffca713d1c6f2d5d5cf696fa417"
server_key_iv: "9504de36e7d01f3120251e3e65092772"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.916594 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "84d6e93366d047178d0a8ad5fc9de195"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "356a2ffca713d1c6f2d5d5cf696fa417"
server_key_iv: "9504de36e7d01f3120251e3e65092772"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.922323 6201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:19:44.926472 6216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:44.928350 6201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:19:44.928611 6201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "84d6e93366d047178d0a8ad5fc9de195"
format_stamp: "Formatted at 2026-09-27 15:19:44 on dist-test-slave-14jq"
server_key: "356a2ffca713d1c6f2d5d5cf696fa417"
server_key_iv: "9504de36e7d01f3120251e3e65092772"
server_key_version: "encryptionkey@0"
I20260927 15:19:44.928846 6201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:44.957499 6201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:44.962348 6201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:44.962680 6201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:44.963879 6201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:44.965547 6201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:44.965658 6201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:44.965833 6201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:44.965953 6201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:19:45.009604 6201 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:34839
I20260927 15:19:45.009724 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:34839 every 8 connection(s)
I20260927 15:19:45.011234 6201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:45.011754 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6201
I20260927 15:19:45.012005 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:45.012960 3033 external_mini_cluster.cc:1428] Setting key 1f4005d68d39fbecd8ffffe543458e3d
Sep 27 15:19:45 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:45.039004 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:45.014544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:36113 (local address 127.2.246.126:43615)
0927 15:19:45.015178 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:45.015185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:45.016356 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 15:19:45.018662 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:45.018668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:45.018793 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:45.018934 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:45.020892 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.021687 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.024770 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.025324 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.025506 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:45.031039 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:45.031045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:45.031049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:45.031089 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:45.034698 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:45.035743 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:45.035754 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:45.035761 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:45.035967 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:45.036633 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:45.036638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:45.036640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:45.036983 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:45.037127 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:45.038175 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 15:19:45.038510 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
I20260927 15:19:45.040391 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:45.015151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43615 (local address 127.2.246.65:36113)
0927 15:19:45.016171 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:45.016227 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:45.018307 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:45.019238 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:45.019249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:45.019928 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:45.020678 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:45.020700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.021915 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.021925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:45.024444 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:45.024470 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.025699 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.025709 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:45.026166 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:45.027187 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:45.027227 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:45.030714 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:45.034912 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:45.034924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:45.034946 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:45.035470 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:45.036135 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:45.036139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:45.036143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:45.036353 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:45.037143 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:45.037150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:45.037667 (+ 517us) client_negotiation.cc:769] Sending connection context
0927 15:19:45.038043 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 15:19:45.038650 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":387,"threads_started":1}
I20260927 15:19:45.042467 6329 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43615
I20260927 15:19:45.042975 6329 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:45.044003 6329 heartbeater.cc:511] Master 127.2.246.126:43615 requested a full tablet report, sending...
I20260927 15:19:45.047019 6140 ts_manager.cc:194] Registered new tserver with Master: 84d6e93366d047178d0a8ad5fc9de195 (127.2.246.65:34839)
I20260927 15:19:45.048765 6140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:36113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:45.070284 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:45.511104 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:45.511579 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:45.511783 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:45.520100 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:45.520331 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:45.520526 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:45.532995 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:45.534924 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:45.536556 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:45.554850 6340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:45.555007 6341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:45.554950 6343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:45.556445 6335 server_base.cc:1034] running on GCE node
I20260927 15:19:45.557287 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:45.558645 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:45.559930 6335 hybrid_clock.cc:648] HybridClock initialized: now 1790522385559868 us; error 67 us; skew 500 ppm
I20260927 15:19:45.560391 6335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:45 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522385, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:45.564461 6335 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:45.566468 6335 webserver.cc:460] Webserver started at http://127.2.246.66:42337/ using document root <none> and password file <none>
I20260927 15:19:45.567335 6335 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:45.567497 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:45.567827 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:45.571062 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "c4902046923445b5a32cb57b0811e828"
format_stamp: "Formatted at 2026-09-27 15:19:45 on dist-test-slave-14jq"
server_key: "2255f0137a02864afd5ff3441a8920c0"
server_key_iv: "1b95fc0160ec9c28bd07fbd07cd44570"
server_key_version: "encryptionkey@0"
I20260927 15:19:45.571904 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "c4902046923445b5a32cb57b0811e828"
format_stamp: "Formatted at 2026-09-27 15:19:45 on dist-test-slave-14jq"
server_key: "2255f0137a02864afd5ff3441a8920c0"
server_key_iv: "1b95fc0160ec9c28bd07fbd07cd44570"
server_key_version: "encryptionkey@0"
I20260927 15:19:45.577589 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:19:45.581738 6350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:45.583981 6335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:19:45.584205 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "c4902046923445b5a32cb57b0811e828"
format_stamp: "Formatted at 2026-09-27 15:19:45 on dist-test-slave-14jq"
server_key: "2255f0137a02864afd5ff3441a8920c0"
server_key_iv: "1b95fc0160ec9c28bd07fbd07cd44570"
server_key_version: "encryptionkey@0"
I20260927 15:19:45.584437 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:45.612197 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:45.617710 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:45.618139 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:45.619361 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:45.621030 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:45.621148 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:45.621309 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:45.621395 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:45.662277 6335 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:39613
I20260927 15:19:45.662317 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:39613 every 8 connection(s)
I20260927 15:19:45.663913 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:45.668215 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6335
I20260927 15:19:45.668434 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:45.668953 3033 external_mini_cluster.cc:1428] Setting key 087fda395028ac60d775d96e30a30aea
Sep 27 15:19:45 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522385, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:45.688298 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:45.667015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:45413 (local address 127.2.246.126:43615)
0927 15:19:45.667635 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:45.667641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:45.669972 (+ 2331us) server_negotiation.cc:407] Connection header received
0927 15:19:45.670588 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:45.670594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:45.670697 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:45.670822 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:45.672590 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.673329 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.675053 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.675482 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.675621 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:45.679851 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:45.679861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:45.679869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:45.679919 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:45.683814 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:45.684883 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:45.684891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:45.684898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:45.685059 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:45.685728 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:45.685735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:45.685741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:45.686190 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:45.686359 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:45.687445 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0927 15:19:45.687746 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":343,"threads_started":1}
I20260927 15:19:45.689412 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:45.667351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43615 (local address 127.2.246.66:45413)
0927 15:19:45.668273 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:45.668322 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:45.670274 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:45.671115 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:45.671128 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:45.671769 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:45.672400 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:45.672418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.673494 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.673499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:45.674899 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:45.674911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:45.675716 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:45.675723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:45.675965 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:45.676694 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:45.676717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:45.679488 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:45.684066 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:45.684077 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:45.684097 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:45.684694 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:45.685221 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:45.685225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:45.685228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:45.685561 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:45.686349 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:45.686357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:45.686946 (+ 589us) client_negotiation.cc:769] Sending connection context
0927 15:19:45.687251 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 15:19:45.687855 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:45.691855 6463 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43615
I20260927 15:19:45.692235 6463 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:45.693198 6463 heartbeater.cc:511] Master 127.2.246.126:43615 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:19:45.695307 6140 ts_manager.cc:194] Registered new tserver with Master: c4902046923445b5a32cb57b0811e828 (127.2.246.66:39613)
I20260927 15:19:45.696146 6140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:45413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:45.721073 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:46.053992 6329 heartbeater.cc:503] Master 127.2.246.126:43615 was elected leader, sending a full tablet report...
W20260927 15:19:46.082800 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:46.083137 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:46.083325 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:46.092319 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:46.092471 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:46.092636 6469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:46.102970 6469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:46.104355 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:46.105705 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:46.120684 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:46.120684 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:46.120798 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:46.122327 6469 server_base.cc:1034] running on GCE node
I20260927 15:19:46.123131 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:46.124428 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:46.125715 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790522386125641 us; error 71 us; skew 500 ppm
I20260927 15:19:46.126121 6469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:46.130039 6469 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:46.131955 6469 webserver.cc:460] Webserver started at http://127.2.246.67:46255/ using document root <none> and password file <none>
I20260927 15:19:46.132740 6469 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:46.132905 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:46.133229 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:46.136531 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "2ad62adfc797450da24774945adbdf9a"
format_stamp: "Formatted at 2026-09-27 15:19:46 on dist-test-slave-14jq"
server_key: "19cebcf58abf60684157746d38a0c0d4"
server_key_iv: "31b8703559c66244d064ba9c7a33d92b"
server_key_version: "encryptionkey@0"
I20260927 15:19:46.137411 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "2ad62adfc797450da24774945adbdf9a"
format_stamp: "Formatted at 2026-09-27 15:19:46 on dist-test-slave-14jq"
server_key: "19cebcf58abf60684157746d38a0c0d4"
server_key_iv: "31b8703559c66244d064ba9c7a33d92b"
server_key_version: "encryptionkey@0"
I20260927 15:19:46.143606 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 15:19:46.148066 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:46.149811 6469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:19:46.150089 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "2ad62adfc797450da24774945adbdf9a"
format_stamp: "Formatted at 2026-09-27 15:19:46 on dist-test-slave-14jq"
server_key: "19cebcf58abf60684157746d38a0c0d4"
server_key_iv: "31b8703559c66244d064ba9c7a33d92b"
server_key_version: "encryptionkey@0"
I20260927 15:19:46.150374 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:46.164657 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:46.169996 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:46.170364 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:46.171535 6469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:46.173205 6469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:46.173301 6469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:46.173445 6469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:46.173516 6469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:46.216156 6469 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:41653
I20260927 15:19:46.216288 6596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:41653 every 8 connection(s)
I20260927 15:19:46.217708 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:46.226321 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6469
I20260927 15:19:46.226536 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:46.227177 3033 external_mini_cluster.cc:1428] Setting key 33e496dfa0954a426b7d5e47128aeafe
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:46.242625 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.221497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38517 (local address 127.2.246.126:43615)
0927 15:19:46.222169 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:46.222177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:46.223224 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 15:19:46.225344 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:46.225351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:46.225468 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:46.225671 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:46.228285 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.229320 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.231178 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.231630 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.231794 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.236183 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:46.236188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:46.236192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:46.236228 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:46.239306 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.240168 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.240173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.240176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.240297 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.240743 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.240748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.240756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.241033 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:46.241197 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:46.241955 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 15:19:46.242194 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260927 15:19:46.243827 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.222159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:43615 (local address 127.2.246.67:38517)
0927 15:19:46.223011 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:46.223065 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:46.224997 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:46.226133 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:46.226147 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:46.227059 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:46.228022 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:46.228045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.229513 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.229521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:46.231006 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:46.231020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.231887 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.231895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:46.232182 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.233058 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:46.233082 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:46.235913 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:46.239511 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:46.239522 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:46.239544 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:46.240021 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:46.240425 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:46.240429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:46.240433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:46.240621 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:46.241160 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:46.241169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:46.241545 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 15:19:46.241825 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 15:19:46.242427 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":347,"threads_started":1}
I20260927 15:19:46.245544 6597 heartbeater.cc:347] Connected to a master server at 127.2.246.126:43615
I20260927 15:19:46.246016 6597 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:46.246956 6597 heartbeater.cc:511] Master 127.2.246.126:43615 requested a full tablet report, sending...
I20260927 15:19:46.248716 6140 ts_manager.cc:194] Registered new tserver with Master: 2ad62adfc797450da24774945adbdf9a (127.2.246.67:41653)
I20260927 15:19:46.249720 6140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:38517
I20260927 15:19:46.257642 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:46.318634 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.301922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.2.246.126:43615)
0927 15:19:46.302157 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:46.302163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:46.302681 (+ 518us) server_negotiation.cc:407] Connection header received
0927 15:19:46.302929 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:46.302935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:46.303042 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:46.303204 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:46.304772 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.305618 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.307130 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.307687 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.307825 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.312372 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:46.312379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:46.312384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:46.312437 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:46.315772 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.316517 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.316522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.316524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.316732 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.317156 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.317159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.317161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.317561 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:46.317762 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:46.318145 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 15:19:46.318350 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:19:46.319381 6140 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49830
I20260927 15:19:46.321421 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6201
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:46.339052 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.322718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.2.246.65:34839)
0927 15:19:46.323430 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:46.323437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:46.323462 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:19:46.323529 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:46.323536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:46.323757 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:46.323941 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:46.325122 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.326199 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.327396 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.328135 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.328288 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.332856 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:46.332869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:46.332884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:46.332957 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:46.335922 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.336779 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.336787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.336789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.336920 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.337341 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.337344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.337347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.337721 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:46.338051 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:46.338451 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:19:46.338687 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":380,"threads_started":1}
W20260927 15:19:46.340801 6324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56308
W20260927 15:19:46.342226 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:46.342491 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6201
I20260927 15:19:46.361001 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6335
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:46.379241 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.362111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.2.246.66:39613)
0927 15:19:46.362723 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:46.362731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:46.362760 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:19:46.362825 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:46.362831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:46.363060 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:46.363204 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:46.364521 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.365600 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.367225 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.367838 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.367972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.372526 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:46.372542 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:46.372560 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:46.372602 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:46.376150 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.377038 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.377045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.377048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.377184 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.377580 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.377584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.377586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.378032 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:46.378272 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:46.378598 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:19:46.378820 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":346,"threads_started":1}
W20260927 15:19:46.380637 6458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52598
W20260927 15:19:46.381826 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6335: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:46.382133 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6335
I20260927 15:19:46.400905 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6469
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522386, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:46.420845 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:46.402262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.2.246.67:41653)
0927 15:19:46.403012 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:46.403019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:46.403051 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:19:46.403120 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:46.403127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:46.403424 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:46.403579 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:46.405166 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.406399 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.407965 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:46.408549 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:46.408662 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:46.414126 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:46.414146 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:46.414166 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:46.414228 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:46.417535 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.418503 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.418513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.418516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.418699 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:46.419150 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:46.419154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:46.419156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:46.419586 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:46.419887 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:46.420221 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:19:46.420480 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":412,"threads_started":1}
W20260927 15:19:46.422371 6592 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34086
W20260927 15:19:46.423599 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:19:46.423895 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6469
I20260927 15:19:46.441423 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6108
I20260927 15:19:46.443014 6170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:46.561327 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6108
2026-09-27T15:19:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4216 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6615](info): set up 2 sockets
Sep 27 15:19:46 dist-test-slave-14jq krb5kdc[6615](info): commencing operation
krb5kdc: starting...
W20260927 15:19:47.682524 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:47 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:47.913547 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34755
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:48.242899 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:48.243251 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:48.243348 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:48.250882 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:48.251045 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:48.251097 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:48.251151 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:48.251204 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:48.261567 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:48.263036 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:48.264425 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:48.279603 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:48.281745 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:48.281859 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:48.282336 6628 server_base.cc:1034] running on GCE node
I20260927 15:19:48.284003 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:48.286563 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:48.290086 6628 hybrid_clock.cc:648] HybridClock initialized: now 1790522388289953 us; error 102 us; skew 500 ppm
I20260927 15:19:48.290632 6628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:48 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:48.296869 6628 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:48.299741 6628 webserver.cc:460] Webserver started at http://127.2.246.126:41329/ using document root <none> and password file <none>
I20260927 15:19:48.301052 6628 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:48.301467 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:48.302030 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:48.308205 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "bfcfb07a83694dcbb387d8d04c6b1bac"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "0fce5f49335a02c5081cbc608aa4a438"
server_key_iv: "edb17dcc3d2243e2ee1add0c934bb77a"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.309795 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "bfcfb07a83694dcbb387d8d04c6b1bac"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "0fce5f49335a02c5081cbc608aa4a438"
server_key_iv: "edb17dcc3d2243e2ee1add0c934bb77a"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.318610 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260927 15:19:48.323740 6643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:48.325680 6628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260927 15:19:48.325930 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "bfcfb07a83694dcbb387d8d04c6b1bac"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "0fce5f49335a02c5081cbc608aa4a438"
server_key_iv: "edb17dcc3d2243e2ee1add0c934bb77a"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.326156 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:48.340085 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:48.345304 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:48.345709 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:48.368371 6628 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:34755
I20260927 15:19:48.368445 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:34755 every 8 connection(s)
I20260927 15:19:48.370631 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:48.375530 6695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:48.378337 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6628
I20260927 15:19:48.378607 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:48.379442 3033 external_mini_cluster.cc:1428] Setting key 25e47563197028ef2236964aa08e8e12
I20260927 15:19:48.390688 6695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac: Bootstrap starting.
I20260927 15:19:48.394835 6695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:48.396288 6695 log.cc:824] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:19:48 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:48.400058 6695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac: No bootstrap required, opened a new log
I20260927 15:19:48.408969 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:48.382459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.2.246.126:34755)
0927 15:19:48.383593 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:48.383609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:48.383667 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:19:48.386010 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:48.386042 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:48.386656 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:48.387132 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:48.389062 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:48.390807 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:48.392807 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:48.393773 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:48.394081 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:48.399557 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:48.399567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:48.399590 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:48.399666 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:48.403490 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:48.404453 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:48.404462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:48.404471 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:48.404786 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:48.405445 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:48.405449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:48.405451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:48.406319 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:48.406569 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:48.407094 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 15:19:48.407662 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":525,"threads_started":1}
I20260927 15:19:48.409164 6695 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } }
I20260927 15:19:48.409653 6695 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:48.409794 6695 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcfb07a83694dcbb387d8d04c6b1bac, State: Initialized, Role: FOLLOWER
I20260927 15:19:48.410549 6695 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } }
I20260927 15:19:48.410851 6695 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:48.411027 6695 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:48.411226 6695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:48.414011 6695 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } }
I20260927 15:19:48.414505 6695 leader_election.cc:302] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfcfb07a83694dcbb387d8d04c6b1bac; no voters:
I20260927 15:19:48.415096 6695 leader_election.cc:288] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:48.415359 6700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:48.416260 6700 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [term 1 LEADER]: Becoming Leader. State: Replica: bfcfb07a83694dcbb387d8d04c6b1bac, State: Running, Role: LEADER
I20260927 15:19:48.417058 6700 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } }
I20260927 15:19:48.417578 6695 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:48.421231 6702 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfcfb07a83694dcbb387d8d04c6b1bac. Latest consensus state: current_term: 1 leader_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } } }
I20260927 15:19:48.421661 6702 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:48.422837 6707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:48.422613 6701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcfb07a83694dcbb387d8d04c6b1bac" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34755 } } }
I20260927 15:19:48.424326 6701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:48.432678 6707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:48.445465 6707 catalog_manager.cc:1347] Generated new cluster ID: 9901038f2688434082042d734dea34c1
I20260927 15:19:48.445627 6707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:48.475334 6707 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:48.477555 6707 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:48.489795 6707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfcfb07a83694dcbb387d8d04c6b1bac: Generated new TSK 0
I20260927 15:19:48.491200 6707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:48.547358 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:48.902606 6721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:48.903047 6721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:48.903246 6721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:48.912425 6721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:48.912627 6721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:48.912825 6721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:48.925101 6721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:48.926990 6721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:48.928692 6721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:48.945478 6726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:48.945721 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:48.946408 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:48.947283 6721 server_base.cc:1034] running on GCE node
I20260927 15:19:48.948298 6721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:48.949894 6721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:48.951197 6721 hybrid_clock.cc:648] HybridClock initialized: now 1790522388951138 us; error 60 us; skew 500 ppm
I20260927 15:19:48.951663 6721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:48 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:48.955842 6721 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:48.957909 6721 webserver.cc:460] Webserver started at http://127.2.246.65:39981/ using document root <none> and password file <none>
I20260927 15:19:48.959097 6721 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:48.959296 6721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:48.959685 6721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:48.963524 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "4af64fef17d240fdb8a259d7ce6d64ce"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "b80992d75dcac499835064db193cddc4"
server_key_iv: "40bb6351625ce3db4c7f151221c90ee0"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.964515 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "4af64fef17d240fdb8a259d7ce6d64ce"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "b80992d75dcac499835064db193cddc4"
server_key_iv: "40bb6351625ce3db4c7f151221c90ee0"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.971311 6721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:19:48.976483 6736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:48.978715 6721 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:19:48.978984 6721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "4af64fef17d240fdb8a259d7ce6d64ce"
format_stamp: "Formatted at 2026-09-27 15:19:48 on dist-test-slave-14jq"
server_key: "b80992d75dcac499835064db193cddc4"
server_key_iv: "40bb6351625ce3db4c7f151221c90ee0"
server_key_version: "encryptionkey@0"
I20260927 15:19:48.979264 6721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:49.007095 6721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:49.012857 6721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:49.013245 6721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:49.014537 6721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:49.016196 6721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:49.016312 6721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:49.016485 6721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:49.016570 6721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:49.060922 6721 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:36891
I20260927 15:19:49.061036 6848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:36891 every 8 connection(s)
I20260927 15:19:49.062594 6721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:49.064775 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6721
I20260927 15:19:49.065021 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:49.065773 3033 external_mini_cluster.cc:1428] Setting key 9223b8fd77e0eeb3a97a4ef13316f7ee
Sep 27 15:19:49 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:49.090873 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:49.066523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:37315 (local address 127.2.246.126:34755)
0927 15:19:49.067299 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:49.067306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:49.068492 (+ 1186us) server_negotiation.cc:407] Connection header received
0927 15:19:49.070833 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:49.070840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:49.070945 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:49.071097 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:49.073151 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.074306 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.076348 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.076815 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.076978 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:49.082405 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:49.082412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:49.082417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:49.082465 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:49.086390 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:49.087517 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:49.087524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:49.087526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:49.087767 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:49.088477 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:49.088485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:49.088489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:49.088982 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:49.089208 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:49.090155 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 15:19:49.090449 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":459,"threads_started":1}
I20260927 15:19:49.091930 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:49.067121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34755 (local address 127.2.246.65:37315)
0927 15:19:49.068245 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:49.068327 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:49.070459 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:49.071409 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:49.071420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:49.072120 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:49.072901 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:49.072924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.074569 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.074574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:49.076156 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:49.076173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.077120 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.077132 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:49.077430 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:49.078403 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:49.078434 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:49.081973 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:49.086648 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:49.086658 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:49.086675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:49.087334 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:49.087942 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:49.087951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:49.087958 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:49.088261 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:49.089156 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:49.089163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:49.089618 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 15:19:49.089991 (+ 373us) client_negotiation.cc:240] Negotiation successful
0927 15:19:49.090596 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":382,"threads_started":1}
I20260927 15:19:49.094403 6849 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34755
I20260927 15:19:49.094801 6849 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:49.095657 6849 heartbeater.cc:511] Master 127.2.246.126:34755 requested a full tablet report, sending...
I20260927 15:19:49.098320 6660 ts_manager.cc:194] Registered new tserver with Master: 4af64fef17d240fdb8a259d7ce6d64ce (127.2.246.65:36891)
I20260927 15:19:49.099925 6660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:37315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:49.121989 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:49.553063 6855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:49.553480 6855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:49.553602 6855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:49.562351 6855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:49.562533 6855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:49.562677 6855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:49.574491 6855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:49.576467 6855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:49.578725 6855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:49.598294 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:49.598308 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:49.598272 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:49.600137 6855 server_base.cc:1034] running on GCE node
I20260927 15:19:49.601285 6855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:49.602782 6855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:49.604100 6855 hybrid_clock.cc:648] HybridClock initialized: now 1790522389604031 us; error 50 us; skew 500 ppm
I20260927 15:19:49.604502 6855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:49 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:49.608933 6855 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:49.611073 6855 webserver.cc:460] Webserver started at http://127.2.246.66:45885/ using document root <none> and password file <none>
I20260927 15:19:49.612098 6855 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:49.612268 6855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:49.612672 6855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:49.616628 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "a1328c18b673456388ef46e6620e2ec0"
format_stamp: "Formatted at 2026-09-27 15:19:49 on dist-test-slave-14jq"
server_key: "48947fda9c632f7f161b2f561a66439d"
server_key_iv: "5f54aa95886912a69e27fceee78f7078"
server_key_version: "encryptionkey@0"
I20260927 15:19:49.617697 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "a1328c18b673456388ef46e6620e2ec0"
format_stamp: "Formatted at 2026-09-27 15:19:49 on dist-test-slave-14jq"
server_key: "48947fda9c632f7f161b2f561a66439d"
server_key_iv: "5f54aa95886912a69e27fceee78f7078"
server_key_version: "encryptionkey@0"
I20260927 15:19:49.624946 6855 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:19:49.630223 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:49.632727 6855 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:19:49.633039 6855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "a1328c18b673456388ef46e6620e2ec0"
format_stamp: "Formatted at 2026-09-27 15:19:49 on dist-test-slave-14jq"
server_key: "48947fda9c632f7f161b2f561a66439d"
server_key_iv: "5f54aa95886912a69e27fceee78f7078"
server_key_version: "encryptionkey@0"
I20260927 15:19:49.633325 6855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:49.658152 6855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:49.663170 6855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:49.663546 6855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:49.664779 6855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:49.666493 6855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:49.666582 6855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:49.666733 6855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:49.666788 6855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:49.708686 6855 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:42353
I20260927 15:19:49.708734 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:42353 every 8 connection(s)
I20260927 15:19:49.710426 6855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:49.711162 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6855
I20260927 15:19:49.711541 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:49.712124 3033 external_mini_cluster.cc:1428] Setting key 62be55f0b64905553c31057c304c69b7
Sep 27 15:19:49 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:49.734977 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:49.714272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:53553 (local address 127.2.246.126:34755)
0927 15:19:49.714871 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:49.714878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:49.716259 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 15:19:49.718597 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:49.718603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:49.718704 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:49.718841 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:49.720895 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.721646 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.723470 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.723925 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.724082 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:49.728205 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:49.728211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:49.728215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:49.728249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:49.731472 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:49.732476 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:49.732486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:49.732488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:49.732604 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:49.733048 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:49.733052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:49.733056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:49.733347 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:49.733517 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:49.734332 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 15:19:49.734613 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:19:49.736138 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:49.715058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34755 (local address 127.2.246.66:53553)
0927 15:19:49.716049 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:49.716108 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:49.718240 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:49.719132 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:49.719143 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:49.719791 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:49.720636 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:49.720667 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.721908 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.721913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:49.723291 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:49.723307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:49.724173 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:49.724181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:49.724437 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:49.725292 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:49.725322 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:49.727941 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:49.731638 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:49.731651 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:49.731674 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:49.732337 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:49.732714 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:49.732720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:49.732724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:49.732922 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:49.733484 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:49.733493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:49.733952 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 15:19:49.734233 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 15:19:49.734783 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":396,"threads_started":1}
I20260927 15:19:49.738161 6983 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34755
I20260927 15:19:49.738668 6983 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:49.739787 6983 heartbeater.cc:511] Master 127.2.246.126:34755 requested a full tablet report, sending...
I20260927 15:19:49.741868 6660 ts_manager.cc:194] Registered new tserver with Master: a1328c18b673456388ef46e6620e2ec0 (127.2.246.66:42353)
I20260927 15:19:49.742789 6660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:49.766558 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:50.105744 6849 heartbeater.cc:503] Master 127.2.246.126:34755 was elected leader, sending a full tablet report...
W20260927 15:19:50.140470 6989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:50.140904 6989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:50.141047 6989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:50.148887 6989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:50.149015 6989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:50.149199 6989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:50.160260 6989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:50.161815 6989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:50.163406 6989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:50.179131 6997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:50.179183 6995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:50.179682 6994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:50.184438 6989 server_base.cc:1034] running on GCE node
I20260927 15:19:50.185308 6989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:50.186615 6989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:50.187907 6989 hybrid_clock.cc:648] HybridClock initialized: now 1790522390187852 us; error 57 us; skew 500 ppm
I20260927 15:19:50.188329 6989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:50.198441 6989 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:50.200603 6989 webserver.cc:460] Webserver started at http://127.2.246.67:46287/ using document root <none> and password file <none>
I20260927 15:19:50.201444 6989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:50.201611 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:50.202040 6989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:50.205518 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "2cd5c9531ac943e39b8441d7cd049c5b"
format_stamp: "Formatted at 2026-09-27 15:19:50 on dist-test-slave-14jq"
server_key: "eb694c2916921d2334ed769aa1eea6c2"
server_key_iv: "f791c9f5c6e608983992a03ff0eeda96"
server_key_version: "encryptionkey@0"
I20260927 15:19:50.206602 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd5c9531ac943e39b8441d7cd049c5b"
format_stamp: "Formatted at 2026-09-27 15:19:50 on dist-test-slave-14jq"
server_key: "eb694c2916921d2334ed769aa1eea6c2"
server_key_iv: "f791c9f5c6e608983992a03ff0eeda96"
server_key_version: "encryptionkey@0"
I20260927 15:19:50.213205 6989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:19:50.217760 7004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:50.220312 6989 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260927 15:19:50.220564 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "2cd5c9531ac943e39b8441d7cd049c5b"
format_stamp: "Formatted at 2026-09-27 15:19:50 on dist-test-slave-14jq"
server_key: "eb694c2916921d2334ed769aa1eea6c2"
server_key_iv: "f791c9f5c6e608983992a03ff0eeda96"
server_key_version: "encryptionkey@0"
I20260927 15:19:50.220858 6989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:50.271029 6989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:50.278940 6989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:50.279438 6989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:50.281354 6989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:50.285056 6989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:50.285223 6989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:50.285405 6989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:50.285466 6989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:50.327704 6989 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:37385
I20260927 15:19:50.327757 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:37385 every 8 connection(s)
I20260927 15:19:50.329325 6989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:50.332655 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 6989
I20260927 15:19:50.332877 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:50.333390 3033 external_mini_cluster.cc:1428] Setting key c14366033cb837091ec75cb08bc48ce8
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:50.355827 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.332719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:47163 (local address 127.2.246.126:34755)
0927 15:19:50.333343 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.333351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.334981 (+ 1630us) server_negotiation.cc:407] Connection header received
0927 15:19:50.337624 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.337630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.337743 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.337987 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.339889 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.340812 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.342747 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.343273 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.343410 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.347581 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.347589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.347594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.347661 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.351783 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.352822 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.352828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.352831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.353007 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.353590 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.353595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.353597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.354035 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.354349 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.355200 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.355444 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260927 15:19:50.357056 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.333616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34755 (local address 127.2.246.67:47163)
0927 15:19:50.334759 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:50.334833 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:50.337153 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:50.338234 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:50.338319 (+ 85us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:50.338985 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:50.339639 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:50.339658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.341033 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.341042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:50.342553 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:50.342569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.343530 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.343535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:50.343758 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.344522 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:50.344546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:50.347230 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:50.351983 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:50.351992 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:50.352010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:50.352587 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:50.353178 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:50.353182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:50.353184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:50.353429 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:50.354306 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:50.354315 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:50.354731 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 15:19:50.355002 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 15:19:50.355481 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":514,"threads_started":1}
I20260927 15:19:50.359867 7117 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34755
I20260927 15:19:50.360352 7117 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:50.361218 7117 heartbeater.cc:511] Master 127.2.246.126:34755 requested a full tablet report, sending...
I20260927 15:19:50.363512 6659 ts_manager.cc:194] Registered new tserver with Master: 2cd5c9531ac943e39b8441d7cd049c5b (127.2.246.67:37385)
I20260927 15:19:50.364799 6659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:47163
I20260927 15:19:50.365206 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:50.426859 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.409934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.2.246.65:36891)
0927 15:19:50.410806 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.410816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.410850 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:19:50.410936 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.410944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.411170 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.411372 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.412768 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.413768 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.415044 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.415618 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.415766 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.420102 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.420118 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.420134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.420174 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.423432 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.424203 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.424211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.424214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.424345 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.424732 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.424737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.424741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.425198 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.425568 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.425938 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.426219 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":440,"threads_started":1}
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:50.449751 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.433602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.2.246.126:34755)
0927 15:19:50.433956 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.433974 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.434247 (+ 273us) server_negotiation.cc:407] Connection header received
0927 15:19:50.434547 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.434554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.434646 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.434825 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.436280 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.437133 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.438791 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.439253 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.439402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.444032 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.444039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.444042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.444084 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.446958 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.447744 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.447757 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.447759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.447884 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.448257 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.448271 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.448273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.448605 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.448858 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.449173 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.449424 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260927 15:19:50.450382 6659 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39108
I20260927 15:19:50.451787 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6721
I20260927 15:19:50.467664 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.452899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.2.246.65:36891)
0927 15:19:50.453102 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.453110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.453543 (+ 433us) server_negotiation.cc:407] Connection header received
0927 15:19:50.453701 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.453705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.453908 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.454027 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.455491 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.456345 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.457801 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.458350 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.458481 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.461377 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.461382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.461385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.461423 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.464349 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.465389 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.465393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.465396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.465558 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.466195 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.466200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.466202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.466569 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.466764 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.467206 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.467415 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:19:50.468537 6844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:50.560204 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6721
I20260927 15:19:50.577409 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6855
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:50.598493 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.578545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.2.246.66:42353)
0927 15:19:50.579159 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.579166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.579196 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:19:50.579262 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.579269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.579493 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.579655 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.581125 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.582363 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.584387 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.585071 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.585223 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.591063 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.591079 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.591096 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.591149 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.594611 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.595660 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.595669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.595671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.595832 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.596396 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.596400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.596402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.596815 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.597078 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.597653 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.598055 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":346,"threads_started":1}
I20260927 15:19:50.599820 6978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:50.690150 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6855
I20260927 15:19:50.708801 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6989
Sep 27 15:19:50 dist-test-slave-14jq krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:50.730491 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:50.710052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.2.246.67:37385)
0927 15:19:50.710805 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:50.710813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:50.710846 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:19:50.710914 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:50.710921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:50.711278 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:50.711445 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:50.713529 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.714697 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.716412 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:50.717123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:50.717339 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:50.723166 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:50.723182 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:50.723199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:50.723261 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:50.726860 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.727972 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.727979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.727982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.728130 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:50.728626 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:50.728630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:50.728632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:50.729046 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:50.729389 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:50.729710 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 15:19:50.730058 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":420,"threads_started":1}
I20260927 15:19:50.731777 7112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:50.827567 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6989
I20260927 15:19:50.844096 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 6628
I20260927 15:19:50.845887 6690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:50.952497 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 6628
2026-09-27T15:19:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4389 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:51 dist-test-slave-14jq krb5kdc[7137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:51 dist-test-slave-14jq krb5kdc[7137](info): set up 2 sockets
Sep 27 15:19:51 dist-test-slave-14jq krb5kdc[7137](info): commencing operation
krb5kdc: starting...
W20260927 15:19:52.078662 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:19:52 dist-test-slave-14jq krb5kdc[7137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:19:52.317759 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34417
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:52.668004 7150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:52.668331 7150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:52.668481 7150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:52.675999 7150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:52.676117 7150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:52.676340 7150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:52.676542 7150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:52.676637 7150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:52.686998 7150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:52.688434 7150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:52.689779 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:52.704613 7156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:52.704777 7158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:52.704631 7155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:52.707662 7150 server_base.cc:1034] running on GCE node
I20260927 15:19:52.708679 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:52.710484 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:52.711790 7150 hybrid_clock.cc:648] HybridClock initialized: now 1790522392711708 us; error 78 us; skew 500 ppm
I20260927 15:19:52.712224 7150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:52 dist-test-slave-14jq krb5kdc[7137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:52.715965 7150 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:19:52.717471 7150 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:52.717610 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:52.718014 7150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:52.721145 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "e2021639c91042c285749204ca043f2c"
format_stamp: "Formatted at 2026-09-27 15:19:52 on dist-test-slave-14jq"
server_key: "c5ca12e18f3ed2447a9760f9c05cdd9f"
server_key_iv: "ae6de92663c3f331ef2424ecc753f528"
server_key_version: "encryptionkey@0"
I20260927 15:19:52.722070 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "e2021639c91042c285749204ca043f2c"
format_stamp: "Formatted at 2026-09-27 15:19:52 on dist-test-slave-14jq"
server_key: "c5ca12e18f3ed2447a9760f9c05cdd9f"
server_key_iv: "ae6de92663c3f331ef2424ecc753f528"
server_key_version: "encryptionkey@0"
I20260927 15:19:52.727896 7150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:19:52.732170 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:52.734081 7150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:52.734335 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "e2021639c91042c285749204ca043f2c"
format_stamp: "Formatted at 2026-09-27 15:19:52 on dist-test-slave-14jq"
server_key: "c5ca12e18f3ed2447a9760f9c05cdd9f"
server_key_iv: "ae6de92663c3f331ef2424ecc753f528"
server_key_version: "encryptionkey@0"
I20260927 15:19:52.734603 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:52.767280 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:52.772341 7150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:52.772655 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:52.794596 7150 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:34417
I20260927 15:19:52.794696 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:34417 every 8 connection(s)
I20260927 15:19:52.795701 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:52.800760 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:52.803110 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7150
I20260927 15:19:52.803390 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:52.804087 3033 external_mini_cluster.cc:1428] Setting key efe038cba514f86e50bd4ad3ea76f7b5
I20260927 15:19:52.817320 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: Bootstrap starting.
Sep 27 15:19:52 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:52.822701 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:52.824070 7216 log.cc:824] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:52.828476 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: No bootstrap required, opened a new log
I20260927 15:19:52.832834 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:52.806594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.2.246.126:34417)
0927 15:19:52.807669 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:52.807682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:52.807729 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:19:52.809667 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:52.809693 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:52.810237 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:52.810616 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:52.813158 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.814781 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.816858 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.817997 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.818185 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:52.823968 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:52.823978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:52.824001 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:52.824068 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:52.827873 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.828726 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.828736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.828752 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.828950 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.829360 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.829365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.829370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.830186 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:52.830530 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:52.831047 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 15:19:52.831572 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":480,"threads_started":1}
I20260927 15:19:52.838378 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } }
I20260927 15:19:52.838909 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:52.839025 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2021639c91042c285749204ca043f2c, State: Initialized, Role: FOLLOWER
I20260927 15:19:52.839744 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } }
I20260927 15:19:52.840036 7216 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:52.840204 7216 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:52.840400 7216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:52.843350 7216 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } }
I20260927 15:19:52.844029 7216 leader_election.cc:302] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2021639c91042c285749204ca043f2c; no voters:
I20260927 15:19:52.844724 7216 leader_election.cc:288] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:52.844965 7221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:52.845904 7221 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [term 1 LEADER]: Becoming Leader. State: Replica: e2021639c91042c285749204ca043f2c, State: Running, Role: LEADER
I20260927 15:19:52.846577 7221 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } }
I20260927 15:19:52.847432 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:52.862040 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2021639c91042c285749204ca043f2c. Latest consensus state: current_term: 1 leader_uuid: "e2021639c91042c285749204ca043f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } } }
I20260927 15:19:52.862694 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:52.866499 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2021639c91042c285749204ca043f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2021639c91042c285749204ca043f2c" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34417 } } }
I20260927 15:19:52.867718 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:52.869100 7229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:52.876362 7229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:52.888351 7229 catalog_manager.cc:1347] Generated new cluster ID: 8984ae19e7414244b057a17b51fe7fee
I20260927 15:19:52.888500 7229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:52.914304 7229 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:52.916501 7229 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:52.928645 7229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: Generated new TSK 0
I20260927 15:19:52.930222 7229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:19:52.997910 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:53.436830 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:53.437204 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:53.437386 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:53.445713 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:53.445959 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:53.446215 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:53.458297 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:53.460089 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:53.461778 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:53.477900 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:53.477955 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:53.477988 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:53.479717 7242 server_base.cc:1034] running on GCE node
I20260927 15:19:53.480413 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:53.481443 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:53.482666 7242 hybrid_clock.cc:648] HybridClock initialized: now 1790522393482568 us; error 91 us; skew 500 ppm
I20260927 15:19:53.482992 7242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:53 dist-test-slave-14jq krb5kdc[7137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:53.486756 7242 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:19:53.488298 7242 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:53.488492 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:53.488853 7242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:53.492195 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "f77bdb70382341cfbfee8a4829d22850"
format_stamp: "Formatted at 2026-09-27 15:19:53 on dist-test-slave-14jq"
server_key: "2f34fad8e1e7958d3643c749bf9650e9"
server_key_iv: "2d6e6656ed5c34173f83dfb9a1f95bb8"
server_key_version: "encryptionkey@0"
I20260927 15:19:53.493105 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "f77bdb70382341cfbfee8a4829d22850"
format_stamp: "Formatted at 2026-09-27 15:19:53 on dist-test-slave-14jq"
server_key: "2f34fad8e1e7958d3643c749bf9650e9"
server_key_iv: "2d6e6656ed5c34173f83dfb9a1f95bb8"
server_key_version: "encryptionkey@0"
I20260927 15:19:53.499204 7242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260927 15:19:53.504021 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.506366 7242 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:53.506649 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "f77bdb70382341cfbfee8a4829d22850"
format_stamp: "Formatted at 2026-09-27 15:19:53 on dist-test-slave-14jq"
server_key: "2f34fad8e1e7958d3643c749bf9650e9"
server_key_iv: "2d6e6656ed5c34173f83dfb9a1f95bb8"
server_key_version: "encryptionkey@0"
I20260927 15:19:53.506925 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:53.543195 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:53.548249 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:53.548604 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:53.549716 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:53.551271 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:53.551405 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.551551 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:53.551643 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.589993 7242 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:40393
I20260927 15:19:53.590109 7368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:40393 every 8 connection(s)
I20260927 15:19:53.590961 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:53.597383 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7242
I20260927 15:19:53.597612 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:53.598290 3033 external_mini_cluster.cc:1428] Setting key 051ed0f2cbcdbfa71c69ed6395bc7ac3
Sep 27 15:19:53 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:53.615965 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.594254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:57191 (local address 127.2.246.126:34417)
0927 15:19:53.594926 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.594932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.596218 (+ 1286us) server_negotiation.cc:407] Connection header received
0927 15:19:53.598768 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.598775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.598897 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.599040 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.601079 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.601919 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.603792 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.604214 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.604364 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.608952 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.608959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.608962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.609004 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.612248 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.613190 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.613196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.613201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.613347 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.613821 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.613828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.613859 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.614188 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.614356 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.615256 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.615559 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:19:53.617414 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.594919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34417 (local address 127.2.246.65:57191)
0927 15:19:53.595966 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.596032 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.598333 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.599356 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.599367 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.600023 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:53.600772 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.600793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.602086 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.602094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.603557 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.603572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.604480 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.604491 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.604773 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.605591 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:53.605631 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:53.608639 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:53.612432 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.612445 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.612473 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.613034 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.613474 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.613480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.613484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.613693 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.614334 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:53.614343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:53.614801 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.615165 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.615865 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":482,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:19:53.620018 7369 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34417
I20260927 15:19:53.620469 7369 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:53.621352 7369 heartbeater.cc:511] Master 127.2.246.126:34417 requested a full tablet report, sending...
I20260927 15:19:53.625128 7181 ts_manager.cc:194] Registered new tserver with Master: f77bdb70382341cfbfee8a4829d22850 (127.2.246.65:40393)
I20260927 15:19:53.627275 7181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:57191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:19:53.650343 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:54.063565 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:54.063946 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:54.064121 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:54.071890 7375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:54.072058 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:54.072250 7375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:54.082388 7375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:54.083913 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:54.085289 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:54.099478 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:54.099628 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:54.100670 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:54.101066 7375 server_base.cc:1034] running on GCE node
I20260927 15:19:54.101955 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:54.103272 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:54.104586 7375 hybrid_clock.cc:648] HybridClock initialized: now 1790522394104490 us; error 76 us; skew 500 ppm
I20260927 15:19:54.105032 7375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:54 dist-test-slave-14jq krb5kdc[7137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:54.109110 7375 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:19:54.110816 7375 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:54.110993 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:54.111418 7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:54.115413 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "199ff6f3318845699e131f2d7e18c09b"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "e2a8680c66cfb82130116269809c258b"
server_key_iv: "f0e0799eb4cbbf480bc48c530ee742ad"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.116458 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "199ff6f3318845699e131f2d7e18c09b"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "e2a8680c66cfb82130116269809c258b"
server_key_iv: "f0e0799eb4cbbf480bc48c530ee742ad"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.123672 7375 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:19:54.131052 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.133242 7375 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:54.133549 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "199ff6f3318845699e131f2d7e18c09b"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "e2a8680c66cfb82130116269809c258b"
server_key_iv: "f0e0799eb4cbbf480bc48c530ee742ad"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.133989 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:54.158224 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:54.164624 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:54.165076 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:54.166577 7375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:54.168609 7375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:54.168732 7375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.168907 7375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:54.169000 7375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.212389 7375 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:42827
I20260927 15:19:54.212432 7501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:42827 every 8 connection(s)
I20260927 15:19:54.213604 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:54.216532 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7375
I20260927 15:19:54.216768 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:54.217322 3033 external_mini_cluster.cc:1428] Setting key c88242264ce5920b1a3b4843aab60fa1
Sep 27 15:19:54 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:19:54.241426 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.216863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:53241 (local address 127.2.246.126:34417)
0927 15:19:54.217447 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.217457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.219026 (+ 1569us) server_negotiation.cc:407] Connection header received
0927 15:19:54.221227 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.221237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.221399 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.221581 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:54.223530 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.224795 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.226849 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.227349 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.227474 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.232696 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:54.232703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:54.232706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:54.232753 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:54.236520 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.237787 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.237792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.237795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.238083 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.238784 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.238795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.238799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.239482 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:54.239748 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.240797 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.241040 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":261,"threads_started":1}
I20260927 15:19:54.242956 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.217701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34417 (local address 127.2.246.66:53241)
0927 15:19:54.218799 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:54.218877 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:54.220864 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:54.221902 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:54.221912 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:54.222523 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:54.223208 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.223230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.225041 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.225045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.226627 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.226642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.227631 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.227642 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.227987 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.229042 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:54.229079 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:54.232320 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:54.236885 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:54.236909 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:54.236930 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:54.237601 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:54.238244 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:54.238249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:54.238253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:54.238551 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:54.239715 (+ 1164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:54.239727 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:54.240264 (+ 537us) client_negotiation.cc:769] Sending connection context
0927 15:19:54.240616 (+ 352us) client_negotiation.cc:240] Negotiation successful
0927 15:19:54.241207 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:19:54.245217 7502 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34417
I20260927 15:19:54.245671 7502 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:54.246649 7502 heartbeater.cc:511] Master 127.2.246.126:34417 requested a full tablet report, sending...
I20260927 15:19:54.248921 7181 ts_manager.cc:194] Registered new tserver with Master: 199ff6f3318845699e131f2d7e18c09b (127.2.246.66:42827)
I20260927 15:19:54.250316 7181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:53241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:19:54.271193 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:54.631893 7369 heartbeater.cc:503] Master 127.2.246.126:34417 was elected leader, sending a full tablet report...
W20260927 15:19:54.639014 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:54.639358 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:54.639472 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:54.647131 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:54.647257 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:54.647374 7508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:54.657516 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:54.658897 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:54.660166 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:54.675206 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:54.675168 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:54.675240 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:54.676508 7508 server_base.cc:1034] running on GCE node
I20260927 15:19:54.677217 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:54.678543 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:54.679816 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790522394679754 us; error 54 us; skew 500 ppm
I20260927 15:19:54.680161 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:19:54 dist-test-slave-14jq krb5kdc[7137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:54.684183 7508 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:19:54.685735 7508 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:54.685916 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:54.686349 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:54.689653 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "28e81552bbdf41b7adfb70cc0cae5b4c"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "caca7e87917e332d5860eaaa802fac35"
server_key_iv: "2f84e4c66982dbe69131c4a775cf5de6"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.690685 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "28e81552bbdf41b7adfb70cc0cae5b4c"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "caca7e87917e332d5860eaaa802fac35"
server_key_iv: "2f84e4c66982dbe69131c4a775cf5de6"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.696918 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:19:54.701371 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.703413 7508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:19:54.703742 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "28e81552bbdf41b7adfb70cc0cae5b4c"
format_stamp: "Formatted at 2026-09-27 15:19:54 on dist-test-slave-14jq"
server_key: "caca7e87917e332d5860eaaa802fac35"
server_key_iv: "2f84e4c66982dbe69131c4a775cf5de6"
server_key_version: "encryptionkey@0"
I20260927 15:19:54.704015 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:54.723076 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:54.728394 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:54.728735 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:54.729866 7508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:54.731570 7508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:54.731729 7508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.731882 7508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:54.731976 7508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:54.775586 7508 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:36113
I20260927 15:19:54.775629 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:36113 every 8 connection(s)
I20260927 15:19:54.776506 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:54.778828 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7508
I20260927 15:19:54.779068 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:54.779721 3033 external_mini_cluster.cc:1428] Setting key e0e054adbb541907724ac080aa05861f
Sep 27 15:19:54 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:19:54.801815 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.779908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:53525 (local address 127.2.246.126:34417)
0927 15:19:54.780449 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.780457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.782510 (+ 2053us) server_negotiation.cc:407] Connection header received
0927 15:19:54.785066 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.785071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.785178 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.785295 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:54.786977 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.787759 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.789363 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.789793 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.789999 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.793910 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:54.793917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:54.793921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:54.793963 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:54.798041 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.799039 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.799049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.799054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.799211 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.799689 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.799696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.799699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.800016 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:54.800196 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.801107 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.801379 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260927 15:19:54.803480 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.780791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34417 (local address 127.2.246.67:53525)
0927 15:19:54.782327 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:54.782384 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:54.784771 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:54.785545 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:54.785555 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:54.786187 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:54.786788 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.786805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.787923 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.787927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.789212 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.789224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.790095 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.790102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.790327 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.791140 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:54.791164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:54.793633 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:54.798246 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:54.798266 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:54.798289 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:54.798892 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:54.799338 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:54.799344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:54.799348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:54.799563 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:54.800168 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:54.800178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:54.800642 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 15:19:54.800983 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 15:19:54.801681 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":738,"threads_started":1}
I20260927 15:19:54.805521 7635 heartbeater.cc:347] Connected to a master server at 127.2.246.126:34417
I20260927 15:19:54.805991 7635 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:54.806823 7635 heartbeater.cc:511] Master 127.2.246.126:34417 requested a full tablet report, sending...
I20260927 15:19:54.809080 7181 ts_manager.cc:194] Registered new tserver with Master: 28e81552bbdf41b7adfb70cc0cae5b4c (127.2.246.67:36113)
I20260927 15:19:54.810341 7181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:53525
I20260927 15:19:54.810798 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:54.831522 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.816545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.2.246.126:34417)
0927 15:19:54.816854 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.816862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.817255 (+ 393us) server_negotiation.cc:407] Connection header received
0927 15:19:54.817502 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.817507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.817621 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.817724 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:54.819407 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.820217 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.822025 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.822509 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.822633 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.825523 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:54.825528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:54.825532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:54.825577 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:54.828589 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.829399 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.829403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.829406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.829545 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:54.829985 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:54.829989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:54.829991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:54.830380 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:54.830587 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.830930 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.831133 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:19:54.836849 7181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:54.845082 7181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:54.885394 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.863212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.2.246.65:40393)
0927 15:19:54.875768 (+ 12556us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.875781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.875825 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:19:54.875955 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.875968 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.876334 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.877487 (+ 1153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:54.877772 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.879492 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.882427 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.884007 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.884178 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.884455 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.884732 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.884934 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12332,"thread_start_us":12139,"threads_started":1}
I20260927 15:19:54.885445 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.861802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40393 (local address 127.0.0.1:60224)
0927 15:19:54.874620 (+ 12818us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:54.874695 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:54.874993 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:54.876682 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:54.876691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:54.876739 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:54.877407 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.877428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.879711 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.879720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.882190 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.882207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.884235 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.884242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.884444 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.884508 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 15:19:54.884690 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 15:19:54.884939 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12475,"mutex_wait_us":28,"thread_start_us":14299,"threads_started":1}
I20260927 15:19:54.886801 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.861886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36113 (local address 127.0.0.1:43902)
0927 15:19:54.876496 (+ 14610us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:54.876525 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:54.876713 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:54.878066 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:54.878071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:54.878095 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:54.878647 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.878662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.880778 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.880784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.883553 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.883570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.885792 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.885801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.886046 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.886090 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:19:54.886189 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:19:54.886342 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14427,"mutex_wait_us":55,"thread_start_us":15171,"threads_started":1}
I20260927 15:19:54.887723 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.862646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.2.246.66:42827)
0927 15:19:54.873602 (+ 10956us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.873617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.877542 (+ 3925us) server_negotiation.cc:407] Connection header received
0927 15:19:54.877946 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.877959 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.878249 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.878483 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:54.879659 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.881064 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.883671 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.885266 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.885408 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.886058 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.886502 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.887335 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10682,"thread_start_us":10991,"threads_started":1}
I20260927 15:19:54.888222 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.862294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:42827 (local address 127.0.0.1:40722)
0927 15:19:54.877390 (+ 15096us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:54.877410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:54.877673 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:54.878887 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:54.878891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:54.878917 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:54.879320 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.879337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.881214 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.881220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.883447 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:54.883460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.886083 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.886089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:54.886291 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.886322 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:19:54.886422 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:19:54.886582 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14966,"mutex_wait_us":8,"thread_start_us":11824,"threads_started":1}
I20260927 15:19:54.888530 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:54.862433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.2.246.67:36113)
0927 15:19:54.876949 (+ 14516us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:54.876961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:54.876999 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:19:54.877125 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:54.877136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:54.877440 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:54.877677 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:54.878880 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.880579 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.883747 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:54.885277 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:54.885408 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:54.885666 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:54.887242 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0927 15:19:54.887499 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14280,"thread_start_us":14119,"threads_started":1}
I20260927 15:19:54.891259 7304 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3ac31f82647478136ecef3e8cb58c (DEFAULT_TABLE table=test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:54.893165 7304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3ac31f82647478136ecef3e8cb58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:54.892590 7570 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3ac31f82647478136ecef3e8cb58c (DEFAULT_TABLE table=test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:54.892509 7437 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3ac31f82647478136ecef3e8cb58c (DEFAULT_TABLE table=test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:54.894013 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3ac31f82647478136ecef3e8cb58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:54.894320 7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3ac31f82647478136ecef3e8cb58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:54.908838 7654 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Bootstrap starting.
I20260927 15:19:54.911763 7656 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Bootstrap starting.
I20260927 15:19:54.913625 7655 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Bootstrap starting.
I20260927 15:19:54.914192 7654 tablet_bootstrap.cc:654] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:54.915544 7654 log.cc:824] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:54.916218 7656 tablet_bootstrap.cc:654] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:54.917699 7656 log.cc:824] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:54.917951 7655 tablet_bootstrap.cc:654] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:54.919409 7655 log.cc:824] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:54.921924 7654 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: No bootstrap required, opened a new log
I20260927 15:19:54.921923 7656 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: No bootstrap required, opened a new log
I20260927 15:19:54.922284 7654 ts_tablet_manager.cc:1397] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260927 15:19:54.922284 7656 ts_tablet_manager.cc:1397] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 15:19:54.922806 7655 tablet_bootstrap.cc:492] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: No bootstrap required, opened a new log
I20260927 15:19:54.923113 7655 ts_tablet_manager.cc:1397] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 15:19:54.931032 7655 raft_consensus.cc:348] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.931464 7655 raft_consensus.cc:374] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:54.931555 7655 raft_consensus.cc:729] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77bdb70382341cfbfee8a4829d22850, State: Initialized, Role: FOLLOWER
I20260927 15:19:54.932211 7655 consensus_queue.cc:260] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.934865 7655 ts_tablet_manager.cc:1428] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260927 15:19:54.935771 7654 raft_consensus.cc:348] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.935771 7656 raft_consensus.cc:348] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.936383 7654 raft_consensus.cc:374] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:54.936383 7656 raft_consensus.cc:374] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:54.936511 7656 raft_consensus.cc:729] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e81552bbdf41b7adfb70cc0cae5b4c, State: Initialized, Role: FOLLOWER
I20260927 15:19:54.936511 7654 raft_consensus.cc:729] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199ff6f3318845699e131f2d7e18c09b, State: Initialized, Role: FOLLOWER
I20260927 15:19:54.937387 7656 consensus_queue.cc:260] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.937390 7654 consensus_queue.cc:260] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:54.940114 7654 ts_tablet_manager.cc:1428] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260927 15:19:54.940285 7635 heartbeater.cc:503] Master 127.2.246.126:34417 was elected leader, sending a full tablet report...
I20260927 15:19:54.940600 7502 heartbeater.cc:503] Master 127.2.246.126:34417 was elected leader, sending a full tablet report...
I20260927 15:19:54.941717 7656 ts_tablet_manager.cc:1428] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260927 15:19:54.966256 7503 tablet.cc:2369] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:55.028164 7636 tablet.cc:2369] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:19:55.094493 7370 tablet.cc:2369] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:55.165228 7662 raft_consensus.cc:482] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:55.165637 7662 raft_consensus.cc:504] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:55.167846 7662 leader_election.cc:288] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28e81552bbdf41b7adfb70cc0cae5b4c (127.2.246.67:36113), f77bdb70382341cfbfee8a4829d22850 (127.2.246.65:40393)
I20260927 15:19:55.179450 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.168326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:55665 (local address 127.2.246.67:36113)
0927 15:19:55.168597 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.168604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.170043 (+ 1439us) server_negotiation.cc:407] Connection header received
0927 15:19:55.170257 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.170262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.170346 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.170452 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:55.171304 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.172561 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.176423 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.177807 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.178055 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.178148 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.178948 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.179146 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:19:55.179453 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.168417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36155 (local address 127.2.246.65:40393)
0927 15:19:55.168706 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.168713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.169667 (+ 954us) server_negotiation.cc:407] Connection header received
0927 15:19:55.170033 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.170039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.170132 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.170257 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:55.171172 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.172562 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.176423 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.177803 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.178055 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.178148 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.178948 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.179146 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 15:19:55.179593 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.168388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36113 (local address 127.2.246.66:55665)
0927 15:19:55.169811 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:55.169901 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:55.170091 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:55.170530 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:55.170537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:55.170556 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:55.171125 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:55.171135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.172795 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.172800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:55.176099 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:55.176122 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.178212 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.178218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:55.178521 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.178586 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 15:19:55.178726 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:19:55.178931 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":294,"thread_start_us":983,"threads_started":1}
I20260927 15:19:55.180063 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.168249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40393 (local address 127.2.246.66:36155)
0927 15:19:55.169449 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:55.169479 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:55.169766 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:55.170290 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:55.170293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:55.170314 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:55.170959 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:55.170981 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.172793 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.172800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:55.176099 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:55.176122 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.178202 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.178211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:55.178517 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.178586 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 15:19:55.178726 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:19:55.178931 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":275,"thread_start_us":757,"threads_started":1}
I20260927 15:19:55.180943 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3ac31f82647478136ecef3e8cb58c" candidate_uuid: "199ff6f3318845699e131f2d7e18c09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" is_pre_election: true
I20260927 15:19:55.181522 7590 raft_consensus.cc:2454] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 199ff6f3318845699e131f2d7e18c09b in term 0.
I20260927 15:19:55.181432 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3ac31f82647478136ecef3e8cb58c" candidate_uuid: "199ff6f3318845699e131f2d7e18c09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77bdb70382341cfbfee8a4829d22850" is_pre_election: true
I20260927 15:19:55.182143 7324 raft_consensus.cc:2454] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 199ff6f3318845699e131f2d7e18c09b in term 0.
I20260927 15:19:55.182432 7392 leader_election.cc:302] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199ff6f3318845699e131f2d7e18c09b, 28e81552bbdf41b7adfb70cc0cae5b4c; no voters:
I20260927 15:19:55.182893 7662 raft_consensus.cc:2781] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:55.183115 7662 raft_consensus.cc:482] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:55.183241 7662 raft_consensus.cc:3037] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:55.185672 7662 raft_consensus.cc:504] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:55.186471 7662 leader_election.cc:288] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [CANDIDATE]: Term 1 election: Requested vote from peers 28e81552bbdf41b7adfb70cc0cae5b4c (127.2.246.67:36113), f77bdb70382341cfbfee8a4829d22850 (127.2.246.65:40393)
I20260927 15:19:55.187114 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3ac31f82647478136ecef3e8cb58c" candidate_uuid: "199ff6f3318845699e131f2d7e18c09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c"
I20260927 15:19:55.187119 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3ac31f82647478136ecef3e8cb58c" candidate_uuid: "199ff6f3318845699e131f2d7e18c09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77bdb70382341cfbfee8a4829d22850"
I20260927 15:19:55.187356 7590 raft_consensus.cc:3037] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:55.187374 7324 raft_consensus.cc:3037] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:55.189818 7324 raft_consensus.cc:2454] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 199ff6f3318845699e131f2d7e18c09b in term 1.
I20260927 15:19:55.189886 7590 raft_consensus.cc:2454] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 199ff6f3318845699e131f2d7e18c09b in term 1.
I20260927 15:19:55.190492 7391 leader_election.cc:302] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199ff6f3318845699e131f2d7e18c09b, f77bdb70382341cfbfee8a4829d22850; no voters:
I20260927 15:19:55.190909 7662 raft_consensus.cc:2781] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:55.191425 7662 raft_consensus.cc:686] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 1 LEADER]: Becoming Leader. State: Replica: 199ff6f3318845699e131f2d7e18c09b, State: Running, Role: LEADER
I20260927 15:19:55.191974 7662 consensus_queue.cc:237] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } }
I20260927 15:19:55.196961 7181 catalog_manager.cc:5495] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b reported cstate change: term changed from 0 to 1, leader changed from <none> to 199ff6f3318845699e131f2d7e18c09b (127.2.246.66). New cstate: current_term: 1 leader_uuid: "199ff6f3318845699e131f2d7e18c09b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "199ff6f3318845699e131f2d7e18c09b" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42827 } health_report { overall_health: HEALTHY } } }
I20260927 15:19:55.262167 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.255789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.2.246.66:42827)
0927 15:19:55.255972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.255977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.256127 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:19:55.256252 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.256256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.256340 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.256521 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:55.256970 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.257767 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.259535 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.259978 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.260109 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.260782 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:55.261442 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:55.261571 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.261779 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.261936 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:19:55.273658 7590 raft_consensus.cc:1261] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 1 FOLLOWER]: Refusing update from remote peer 199ff6f3318845699e131f2d7e18c09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:55.274206 7324 raft_consensus.cc:1261] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 1 FOLLOWER]: Refusing update from remote peer 199ff6f3318845699e131f2d7e18c09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:55.274871 7665 consensus_queue.cc:1035] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28e81552bbdf41b7adfb70cc0cae5b4c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:55.275522 7662 consensus_queue.cc:1035] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77bdb70382341cfbfee8a4829d22850" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:55.290280 7668 mvcc.cc:205] Tried to move back new op lower bound from 7333979731029323776 to 7333979730710507520. Current Snapshot: MvccSnapshot[applied={T|T < 7333979731029323776}]
I20260927 15:19:55.309015 7181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:55.309536 7181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:55.314800 7181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2021639c91042c285749204ca043f2c: Sending DeleteTablet for 3 replicas of tablet b2a3ac31f82647478136ecef3e8cb58c
I20260927 15:19:55.316430 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3ac31f82647478136ecef3e8cb58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:55 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:43902
I20260927 15:19:55.316443 7304 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3ac31f82647478136ecef3e8cb58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:55 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:60224
I20260927 15:19:55.316990 7437 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3ac31f82647478136ecef3e8cb58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:55 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:40722
I20260927 15:19:55.323117 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7242
I20260927 15:19:55.327116 7680 tablet_replica.cc:330] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: stopping tablet replica
I20260927 15:19:55.327378 7679 tablet_replica.cc:330] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: stopping tablet replica
I20260927 15:19:55.327935 7679 raft_consensus.cc:2229] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:55.328490 7679 raft_consensus.cc:2258] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:55.328807 7680 raft_consensus.cc:2229] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:55.329571 7680 raft_consensus.cc:2258] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:55.331072 7679 ts_tablet_manager.cc:1905] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:55.331833 7678 tablet_replica.cc:330] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: stopping tablet replica
I20260927 15:19:55.332296 7680 ts_tablet_manager.cc:1905] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:55.332558 7678 raft_consensus.cc:2229] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:55.333150 7678 raft_consensus.cc:2258] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:55.335528 7678 ts_tablet_manager.cc:1905] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:55.339880 7679 ts_tablet_manager.cc:1918] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:55.340173 7679 log.cc:1196] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/b2a3ac31f82647478136ecef3e8cb58c
I20260927 15:19:55.340762 7679 ts_tablet_manager.cc:1939] T b2a3ac31f82647478136ecef3e8cb58c P f77bdb70382341cfbfee8a4829d22850: Deleting consensus metadata
I20260927 15:19:55.341950 7680 ts_tablet_manager.cc:1918] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:55.342171 7680 log.cc:1196] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/b2a3ac31f82647478136ecef3e8cb58c
I20260927 15:19:55.342870 7680 ts_tablet_manager.cc:1939] T b2a3ac31f82647478136ecef3e8cb58c P 199ff6f3318845699e131f2d7e18c09b: Deleting consensus metadata
I20260927 15:19:55.343202 7166 catalog_manager.cc:4841] TS f77bdb70382341cfbfee8a4829d22850 (127.2.246.65:40393): tablet b2a3ac31f82647478136ecef3e8cb58c (table test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]) successfully deleted
I20260927 15:19:55.344198 7678 ts_tablet_manager.cc:1918] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:55.344485 7678 log.cc:1196] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/b2a3ac31f82647478136ecef3e8cb58c
I20260927 15:19:55.344971 7168 catalog_manager.cc:4841] TS 199ff6f3318845699e131f2d7e18c09b (127.2.246.66:42827): tablet b2a3ac31f82647478136ecef3e8cb58c (table test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]) successfully deleted
I20260927 15:19:55.345144 7678 ts_tablet_manager.cc:1939] T b2a3ac31f82647478136ecef3e8cb58c P 28e81552bbdf41b7adfb70cc0cae5b4c: Deleting consensus metadata
I20260927 15:19:55.347329 7167 catalog_manager.cc:4841] TS 28e81552bbdf41b7adfb70cc0cae5b4c (127.2.246.67:36113): tablet b2a3ac31f82647478136ecef3e8cb58c (table test-table [id=79f86da6bc3f4f6f9e254adbecafc0ac]) successfully deleted
Sep 27 15:19:55 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:19:55.373193 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.324695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.2.246.65:40393)
0927 15:19:55.324944 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.324951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.340931 (+ 15980us) server_negotiation.cc:407] Connection header received
0927 15:19:55.344129 (+ 3198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.344135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.344248 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.344329 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:55.346535 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.347846 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.349349 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.350095 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.350270 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.364108 (+ 13838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:55.364135 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:55.364159 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:55.364229 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:55.368841 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.370073 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.370089 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.370096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.370333 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.371031 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.371037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.371039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.371736 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:55.372071 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.372503 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.372763 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:19:55.374100 7364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:55.476969 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7242
I20260927 15:19:55.497258 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7375
Sep 27 15:19:55 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:19:55.520720 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.498272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.2.246.66:42827)
0927 15:19:55.498478 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.498485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.498770 (+ 285us) server_negotiation.cc:407] Connection header received
0927 15:19:55.498997 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.499001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.499083 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.499194 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:55.501126 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.502344 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.504629 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.505275 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.505422 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.511957 (+ 6535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:55.511984 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:55.512010 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:55.512075 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:55.516897 (+ 4822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.518143 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.518154 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.518159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.518337 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.518779 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.518784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.518789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.519322 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:55.519702 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.519974 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.520245 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:19:55.521746 7497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:55.644012 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7375
I20260927 15:19:55.663805 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7508
Sep 27 15:19:55 dist-test-slave-14jq krb5kdc[7137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:19:55.681427 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:55.664782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.2.246.67:36113)
0927 15:19:55.665030 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:55.665037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:55.665154 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:19:55.665411 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:55.665415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:55.665503 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:55.665619 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:55.667083 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.667805 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.669127 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:55.669548 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:55.669683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:55.675279 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:55.675297 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:55.675314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:55.675355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:55.678540 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.679482 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.679490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.679492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.679647 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:55.680020 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:55.680023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:55.680025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:55.680402 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:55.680638 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:55.680929 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:19:55.681102 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:19:55.682354 7630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:55.784194 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7508
I20260927 15:19:55.801577 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7150
I20260927 15:19:55.803061 7211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:55.917333 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7150
2026-09-27T15:19:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4968 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:55Z Disabled control of system clock
I20260927 15:19:55.994545 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44595
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:19:56.328027 7689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:56.328352 7689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:56.328492 7689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:56.335975 7689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:56.336102 7689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:56.336141 7689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:56.336225 7689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:56.336310 7689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:56.345981 7689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:56.347355 7689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:56.349166 7689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:56.362569 7694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:56.362573 7695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:56.362677 7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:56.364035 7689 server_base.cc:1034] running on GCE node
I20260927 15:19:56.365095 7689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:56.367087 7689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:56.368386 7689 hybrid_clock.cc:648] HybridClock initialized: now 1790522396368319 us; error 55 us; skew 500 ppm
I20260927 15:19:56.368877 7689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:19:56.372195 7689 webserver.cc:460] Webserver started at http://127.2.246.126:38219/ using document root <none> and password file <none>
I20260927 15:19:56.373199 7689 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:56.373365 7689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:56.373813 7689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:56.377653 7689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "b41868a247f6422f9c466b48bd9fefb5"
format_stamp: "Formatted at 2026-09-27 15:19:56 on dist-test-slave-14jq"
server_key: "e7a18fdd2c31d19f6d93869661e7dade"
server_key_iv: "4c2a14c67a997063b4c11c8c366b974f"
server_key_version: "encryptionkey@0"
I20260927 15:19:56.378780 7689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "b41868a247f6422f9c466b48bd9fefb5"
format_stamp: "Formatted at 2026-09-27 15:19:56 on dist-test-slave-14jq"
server_key: "e7a18fdd2c31d19f6d93869661e7dade"
server_key_iv: "4c2a14c67a997063b4c11c8c366b974f"
server_key_version: "encryptionkey@0"
I20260927 15:19:56.385984 7689 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 15:19:56.390588 7703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:56.392570 7689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:19:56.392879 7689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "b41868a247f6422f9c466b48bd9fefb5"
format_stamp: "Formatted at 2026-09-27 15:19:56 on dist-test-slave-14jq"
server_key: "e7a18fdd2c31d19f6d93869661e7dade"
server_key_iv: "4c2a14c67a997063b4c11c8c366b974f"
server_key_version: "encryptionkey@0"
I20260927 15:19:56.393186 7689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:56.400497 7689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:56.401602 7689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:56.401996 7689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:56.425138 7689 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44595
I20260927 15:19:56.425155 7754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44595 every 8 connection(s)
I20260927 15:19:56.426998 7689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:56.431612 7755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:56.435798 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7689
I20260927 15:19:56.436002 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:56.436798 3033 external_mini_cluster.cc:1428] Setting key cd8ba5f7061bfbb547b9acbc4bcdf0f4
I20260927 15:19:56.444763 7755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: Bootstrap starting.
I20260927 15:19:56.448616 7755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:56.448845 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:56.439854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.2.246.126:44595)
0927 15:19:56.441488 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:56.441509 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:56.441578 (+ 69us) server_negotiation.cc:407] Connection header received
0927 15:19:56.443816 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:56.443855 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:56.444562 (+ 707us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:56.445149 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:56.446677 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:56.446684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:56.446701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:56.446718 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:56.446794 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:56.446819 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:56.447123 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:56.447288 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:19:56.447630 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":580,"threads_started":1}
I20260927 15:19:56.450119 7755 log.cc:824] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:56.454038 7755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: No bootstrap required, opened a new log
I20260927 15:19:56.461568 7755 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } }
I20260927 15:19:56.462020 7755 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:56.462095 7755 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41868a247f6422f9c466b48bd9fefb5, State: Initialized, Role: FOLLOWER
I20260927 15:19:56.462695 7755 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } }
I20260927 15:19:56.462921 7755 raft_consensus.cc:388] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:56.463014 7755 raft_consensus.cc:482] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:56.463147 7755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:56.465314 7755 raft_consensus.cc:504] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } }
I20260927 15:19:56.465808 7755 leader_election.cc:302] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41868a247f6422f9c466b48bd9fefb5; no voters:
I20260927 15:19:56.466640 7755 leader_election.cc:288] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:56.466868 7760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:56.467988 7760 raft_consensus.cc:686] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [term 1 LEADER]: Becoming Leader. State: Replica: b41868a247f6422f9c466b48bd9fefb5, State: Running, Role: LEADER
I20260927 15:19:56.468789 7760 consensus_queue.cc:237] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } }
I20260927 15:19:56.469244 7755 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:56.472800 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b41868a247f6422f9c466b48bd9fefb5. Latest consensus state: current_term: 1 leader_uuid: "b41868a247f6422f9c466b48bd9fefb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } } }
I20260927 15:19:56.473224 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:56.474004 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b41868a247f6422f9c466b48bd9fefb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41868a247f6422f9c466b48bd9fefb5" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44595 } } }
I20260927 15:19:56.474380 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:56.474843 7767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:56.483845 7767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:56.500905 7767 catalog_manager.cc:1347] Generated new cluster ID: 18cddcf52d5047eeba9ca5fcbd9bd3d6
I20260927 15:19:56.501052 7767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:56.542696 7767 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:56.543443 7767 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:56.552388 7767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: Generated new TSK 0
I20260927 15:19:56.553635 7767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:19:56.563985 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--builtin_ntp_servers=127.2.246.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:19:56.980760 7779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:56.981145 7779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:56.981328 7779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:56.989805 7779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:56.989971 7779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:56.990187 7779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:19:57.002827 7779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:57.004676 7779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:57.007477 7779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:57.027153 7784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:57.027084 7787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:57.027035 7785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:57.028659 7779 server_base.cc:1034] running on GCE node
I20260927 15:19:57.029489 7779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:57.030763 7779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:57.032012 7779 hybrid_clock.cc:648] HybridClock initialized: now 1790522397031952 us; error 56 us; skew 500 ppm
I20260927 15:19:57.032423 7779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:19:57.035511 7779 webserver.cc:460] Webserver started at http://127.2.246.65:36893/ using document root <none> and password file <none>
I20260927 15:19:57.036330 7779 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:57.036432 7779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:57.036814 7779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:57.040064 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "15ac8c1953ad425799afb247c10e3bc3"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "e8e8c762f14dae6945c906f297c38f58"
server_key_iv: "a1211a5a82bd4f810ec27a5326e4f8cd"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.041285 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "15ac8c1953ad425799afb247c10e3bc3"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "e8e8c762f14dae6945c906f297c38f58"
server_key_iv: "a1211a5a82bd4f810ec27a5326e4f8cd"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.049137 7779 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:19:57.054790 7793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.057317 7779 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:57.057610 7779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "15ac8c1953ad425799afb247c10e3bc3"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "e8e8c762f14dae6945c906f297c38f58"
server_key_iv: "a1211a5a82bd4f810ec27a5326e4f8cd"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.058164 7779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:57.067013 7779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:57.068341 7779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:57.068742 7779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:57.070451 7779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:57.072518 7779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:57.072688 7779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.072863 7779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:57.072956 7779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.115593 7779 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:39921
I20260927 15:19:57.115615 7905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:39921 every 8 connection(s)
I20260927 15:19:57.117547 7779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:57.123055 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7779
I20260927 15:19:57.123281 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:57.124126 3033 external_mini_cluster.cc:1428] Setting key c2c2ed48db6784436fe32cd8bde9a572
I20260927 15:19:57.128104 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--builtin_ntp_servers=127.2.246.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:19:57.129290 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:57.121192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:58837 (local address 127.2.246.126:44595)
0927 15:19:57.121658 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:57.121663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:57.124002 (+ 2339us) server_negotiation.cc:407] Connection header received
0927 15:19:57.126245 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:57.126252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:57.126322 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:57.126486 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:57.128364 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:57.128373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:57.128381 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:57.128415 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:57.128474 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:57.128491 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:57.128646 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:57.128886 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:19:57.128949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260927 15:19:57.130839 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:57.122039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44595 (local address 127.2.246.65:58837)
0927 15:19:57.123718 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:57.123784 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:57.125822 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:57.126777 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:57.126787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:57.126930 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:57.127880 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:57.127885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:57.127945 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:57.128076 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:57.128629 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:57.128643 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:57.128654 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:19:57.128985 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 15:19:57.129327 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":488,"threads_started":1}
I20260927 15:19:57.133088 7906 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44595
I20260927 15:19:57.133448 7906 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:57.134240 7906 heartbeater.cc:511] Master 127.2.246.126:44595 requested a full tablet report, sending...
I20260927 15:19:57.137519 7720 ts_manager.cc:194] Registered new tserver with Master: 15ac8c1953ad425799afb247c10e3bc3 (127.2.246.65:39921)
W20260927 15:19:57.564085 7910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:57.564515 7910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:57.564653 7910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:57.573817 7910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:57.574051 7910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:57.574198 7910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:19:57.587594 7910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:57.589394 7910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:57.591843 7910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:57.609192 7915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:57.609474 7916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:57.609660 7918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:57.611286 7910 server_base.cc:1034] running on GCE node
I20260927 15:19:57.612154 7910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:57.613425 7910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:57.614724 7910 hybrid_clock.cc:648] HybridClock initialized: now 1790522397614666 us; error 53 us; skew 500 ppm
I20260927 15:19:57.615149 7910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:19:57.618515 7910 webserver.cc:460] Webserver started at http://127.2.246.66:32887/ using document root <none> and password file <none>
I20260927 15:19:57.619387 7910 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:57.619495 7910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:57.619839 7910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:57.623320 7910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "b5d111b0c0d941cbb0278f7bc04f1e54"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "0f68e99dd088f8849620cd31047d2842"
server_key_iv: "03e27020b6f38ccb636bd5322901141f"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.624214 7910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "b5d111b0c0d941cbb0278f7bc04f1e54"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "0f68e99dd088f8849620cd31047d2842"
server_key_iv: "03e27020b6f38ccb636bd5322901141f"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.630925 7910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 15:19:57.636879 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.639605 7910 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 15:19:57.639948 7910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "b5d111b0c0d941cbb0278f7bc04f1e54"
format_stamp: "Formatted at 2026-09-27 15:19:57 on dist-test-slave-14jq"
server_key: "0f68e99dd088f8849620cd31047d2842"
server_key_iv: "03e27020b6f38ccb636bd5322901141f"
server_key_version: "encryptionkey@0"
I20260927 15:19:57.640292 7910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:57.647631 7910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:57.648788 7910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:57.649123 7910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:57.650441 7910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:57.652827 7910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:57.652994 7910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.653164 7910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:57.653261 7910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:57.693395 7910 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:40255
I20260927 15:19:57.693459 8036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:40255 every 8 connection(s)
I20260927 15:19:57.695293 7910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:57.699908 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 7910
I20260927 15:19:57.700089 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:57.700734 3033 external_mini_cluster.cc:1428] Setting key 2542c3b7faa2d2aebc0ae71b2e570268
I20260927 15:19:57.704530 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--builtin_ntp_servers=127.2.246.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:19:57.706768 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:57.699026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:48485 (local address 127.2.246.126:44595)
0927 15:19:57.699486 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:57.699491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:57.701490 (+ 1999us) server_negotiation.cc:407] Connection header received
0927 15:19:57.703541 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:57.703546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:57.703609 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:57.703766 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:57.705749 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:57.705755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:57.705759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:57.705784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:57.705925 (+ 141us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:57.705951 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:57.706125 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:57.706398 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:19:57.706473 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":260,"threads_started":1}
I20260927 15:19:57.708060 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:57.699819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44595 (local address 127.2.246.66:48485)
0927 15:19:57.701281 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:57.701341 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:57.703171 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:57.704055 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:57.704067 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:57.704210 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:57.705294 (+ 1084us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:57.705300 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:57.705361 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:57.705487 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:57.706088 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:57.706101 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:57.706113 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:19:57.706431 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 15:19:57.706725 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":453,"threads_started":1}
I20260927 15:19:57.710294 8037 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44595
I20260927 15:19:57.710716 8037 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:57.711606 8037 heartbeater.cc:511] Master 127.2.246.126:44595 requested a full tablet report, sending...
I20260927 15:19:57.713591 7720 ts_manager.cc:194] Registered new tserver with Master: b5d111b0c0d941cbb0278f7bc04f1e54 (127.2.246.66:40255)
W20260927 15:19:58.138942 8041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:58.139293 8041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:58.139452 8041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:19:58.142648 7906 heartbeater.cc:503] Master 127.2.246.126:44595 was elected leader, sending a full tablet report...
W20260927 15:19:58.150013 8041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:58.150171 8041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:58.150357 8041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:19:58.160185 8041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:58.161680 8041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:58.163866 8041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:58.179672 8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:58.179764 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:58.179867 8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:58.181514 8041 server_base.cc:1034] running on GCE node
I20260927 15:19:58.182590 8041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:58.184016 8041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:58.185375 8041 hybrid_clock.cc:648] HybridClock initialized: now 1790522398185300 us; error 66 us; skew 500 ppm
I20260927 15:19:58.185931 8041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:19:58.189338 8041 webserver.cc:460] Webserver started at http://127.2.246.67:36425/ using document root <none> and password file <none>
I20260927 15:19:58.190522 8041 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:58.190713 8041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:58.191125 8041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:58.195374 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "98f473ccc6944426a165a2081fd3b4a8"
format_stamp: "Formatted at 2026-09-27 15:19:58 on dist-test-slave-14jq"
server_key: "950d7088a3ecf8b8a9f8de3b5aa69422"
server_key_iv: "c4e34f2c9f658525bfedf2d747259a82"
server_key_version: "encryptionkey@0"
I20260927 15:19:58.197042 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "98f473ccc6944426a165a2081fd3b4a8"
format_stamp: "Formatted at 2026-09-27 15:19:58 on dist-test-slave-14jq"
server_key: "950d7088a3ecf8b8a9f8de3b5aa69422"
server_key_iv: "c4e34f2c9f658525bfedf2d747259a82"
server_key_version: "encryptionkey@0"
I20260927 15:19:58.205709 8041 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260927 15:19:58.211759 8055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:58.214377 8041 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260927 15:19:58.214694 8041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "98f473ccc6944426a165a2081fd3b4a8"
format_stamp: "Formatted at 2026-09-27 15:19:58 on dist-test-slave-14jq"
server_key: "950d7088a3ecf8b8a9f8de3b5aa69422"
server_key_iv: "c4e34f2c9f658525bfedf2d747259a82"
server_key_version: "encryptionkey@0"
I20260927 15:19:58.215039 8041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:58.222464 8041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:58.223948 8041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:58.224334 8041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:58.225550 8041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:58.227367 8041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:58.227464 8041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:58.227573 8041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:58.227632 8041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:58.268875 8041 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:43507
I20260927 15:19:58.269016 8167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:43507 every 8 connection(s)
I20260927 15:19:58.270888 8041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:58.274552 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8041
I20260927 15:19:58.274801 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:58.275494 3033 external_mini_cluster.cc:1428] Setting key bf275aa289c6d29283d2f411708cbe08
I20260927 15:19:58.287707 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.275311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44595 (local address 127.2.246.67:47065)
0927 15:19:58.279858 (+ 4547us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.279917 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.282376 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.283535 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.283550 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.283712 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.284685 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.284689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.284751 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.284968 (+ 217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.285630 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.285642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.285653 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.286047 (+ 394us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.286474 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3840,"thread_start_us":3535,"threads_started":1}
I20260927 15:19:58.288545 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.274524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:47065 (local address 127.2.246.126:44595)
0927 15:19:58.278859 (+ 4335us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.278868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.281936 (+ 3068us) server_negotiation.cc:407] Connection header received
0927 15:19:58.282767 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.282774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.282865 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.283030 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.285317 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.285328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.285334 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.285363 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.285453 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.285473 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.285639 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.288192 (+ 2553us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.288266 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4047,"thread_start_us":3807,"threads_started":1}
I20260927 15:19:58.290190 8168 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44595
I20260927 15:19:58.290555 8168 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:58.291229 8168 heartbeater.cc:511] Master 127.2.246.126:44595 requested a full tablet report, sending...
I20260927 15:19:58.293269 7720 ts_manager.cc:194] Registered new tserver with Master: 98f473ccc6944426a165a2081fd3b4a8 (127.2.246.67:43507)
I20260927 15:19:58.296869 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:58.305665 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.302503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.2.246.126:44595)
0927 15:19:58.302773 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.302779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.303340 (+ 561us) server_negotiation.cc:407] Connection header received
0927 15:19:58.303467 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.303471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.303540 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.303640 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.304910 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.304916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.304920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.304942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.304991 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.305005 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.305169 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.305330 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.305400 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 15:19:58.311288 7720 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:58.319594 7720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:58.351662 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.336151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40255 (local address 127.0.0.1:46648)
0927 15:19:58.344290 (+ 8139us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.344379 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.344694 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.349392 (+ 4698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.349399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.349423 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.350234 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.350240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.350256 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.350294 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.350806 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.350818 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.350827 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.351051 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.351151 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7616,"thread_start_us":7324,"threads_started":1}
I20260927 15:19:58.352021 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.336361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.2.246.66:40255)
0927 15:19:58.346401 (+ 10040us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.346412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.346459 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:19:58.346549 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.346559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.346807 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.347006 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.350464 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.350476 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.350497 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.350520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.350591 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.350623 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.350754 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.351445 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.351597 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9860,"thread_start_us":9713,"threads_started":1}
I20260927 15:19:58.352475 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.337333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:43507 (local address 127.0.0.1:35172)
0927 15:19:58.347110 (+ 9777us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.347131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.347236 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.348772 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.348781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.348896 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.350053 (+ 1157us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.350059 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.350096 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.350190 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.351327 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.351342 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.351349 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.351469 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.351523 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9591,"spinlock_wait_cycles":17664,"thread_start_us":9557,"threads_started":1}
I20260927 15:19:58.353580 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.337076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.2.246.67:43507)
0927 15:19:58.348189 (+ 11113us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.348209 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.348246 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:19:58.348324 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.348347 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.348555 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.349473 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.351076 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.351084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.351097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.351112 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.351157 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.351184 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.352964 (+ 1780us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.353039 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.353152 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10912,"thread_start_us":10765,"threads_started":1}
I20260927 15:19:58.355352 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.348357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44174)
0927 15:19:58.350973 (+ 2616us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.351000 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.351158 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.352277 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.352282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.352316 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.354158 (+ 1842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.354161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.354180 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.354229 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.354713 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.354720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.354727 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.354886 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.354950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1636,"thread_start_us":1825,"threads_started":1}
I20260927 15:19:58.355397 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.348335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.2.246.65:39921)
0927 15:19:58.351034 (+ 2699us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.351047 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.351093 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:19:58.351349 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.351361 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.351655 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.351971 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.354430 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.354440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.354456 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.354476 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.354546 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.354570 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.354795 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.354874 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.354993 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":2321,"threads_started":1}
I20260927 15:19:58.358474 8103 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 (DEFAULT_TABLE table=test-table [id=eeda04c2de69446a983a21af2514f3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:58.359926 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7ef6838afb47ee8cab32f163109e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:58.357717 7972 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 (DEFAULT_TABLE table=test-table [id=eeda04c2de69446a983a21af2514f3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:58.361436 7841 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 (DEFAULT_TABLE table=test-table [id=eeda04c2de69446a983a21af2514f3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:58.362347 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7ef6838afb47ee8cab32f163109e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:58.363132 7841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7ef6838afb47ee8cab32f163109e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:58.374006 8187 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Bootstrap starting.
I20260927 15:19:58.377900 8187 tablet_bootstrap.cc:654] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:58.379123 8187 log.cc:824] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:58.382205 8189 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Bootstrap starting.
I20260927 15:19:58.382561 8187 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: No bootstrap required, opened a new log
I20260927 15:19:58.382865 8187 ts_tablet_manager.cc:1397] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 15:19:58.383311 8190 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: Bootstrap starting.
I20260927 15:19:58.387646 8190 tablet_bootstrap.cc:654] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:58.387739 8189 tablet_bootstrap.cc:654] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:58.389073 8190 log.cc:824] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:58.389165 8189 log.cc:824] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:58.393153 8187 raft_consensus.cc:348] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.393551 8187 raft_consensus.cc:374] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:58.393620 8187 raft_consensus.cc:729] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f473ccc6944426a165a2081fd3b4a8, State: Initialized, Role: FOLLOWER
I20260927 15:19:58.393515 8190 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: No bootstrap required, opened a new log
I20260927 15:19:58.393771 8189 tablet_bootstrap.cc:492] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: No bootstrap required, opened a new log
I20260927 15:19:58.394162 8189 ts_tablet_manager.cc:1397] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260927 15:19:58.394162 8190 ts_tablet_manager.cc:1397] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260927 15:19:58.394413 8187 consensus_queue.cc:260] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.395615 8168 heartbeater.cc:503] Master 127.2.246.126:44595 was elected leader, sending a full tablet report...
I20260927 15:19:58.396344 8187 ts_tablet_manager.cc:1428] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260927 15:19:58.408004 8190 raft_consensus.cc:348] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.408088 8189 raft_consensus.cc:348] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.408726 8189 raft_consensus.cc:374] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:58.408726 8190 raft_consensus.cc:374] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:58.408854 8190 raft_consensus.cc:729] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ac8c1953ad425799afb247c10e3bc3, State: Initialized, Role: FOLLOWER
I20260927 15:19:58.408864 8189 raft_consensus.cc:729] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d111b0c0d941cbb0278f7bc04f1e54, State: Initialized, Role: FOLLOWER
I20260927 15:19:58.409736 8189 consensus_queue.cc:260] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.409736 8190 consensus_queue.cc:260] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.412569 8037 heartbeater.cc:503] Master 127.2.246.126:44595 was elected leader, sending a full tablet report...
I20260927 15:19:58.412802 8189 ts_tablet_manager.cc:1428] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260927 15:19:58.412801 8190 ts_tablet_manager.cc:1428] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
W20260927 15:19:58.448338 8038 tablet.cc:2369] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:58.465474 8195 raft_consensus.cc:482] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:58.465854 8195 raft_consensus.cc:504] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.467730 8195 leader_election.cc:288] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f473ccc6944426a165a2081fd3b4a8 (127.2.246.67:43507), 15ac8c1953ad425799afb247c10e3bc3 (127.2.246.65:39921)
I20260927 15:19:58.472474 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.468691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:47495 (local address 127.2.246.67:43507)
0927 15:19:58.469013 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.469024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.469659 (+ 635us) server_negotiation.cc:407] Connection header received
0927 15:19:58.469757 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.469763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.469881 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.470028 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.471767 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.471772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.471776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.471793 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.471826 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.471838 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.472077 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.472129 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.472193 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:19:58.472669 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.468429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:43507 (local address 127.2.246.66:47495)
0927 15:19:58.469454 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.469475 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.469639 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.470132 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.470140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.470176 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.471468 (+ 1292us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.471472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.471491 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.471541 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.471985 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.471990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.471995 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.472144 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.472195 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":670,"threads_started":1}
I20260927 15:19:58.473585 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.470104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:51063 (local address 127.2.246.65:39921)
0927 15:19:58.470481 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.470487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.470959 (+ 472us) server_negotiation.cc:407] Connection header received
0927 15:19:58.471076 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.471086 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.471190 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.471342 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.472709 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.472717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.472721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.472758 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.472806 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.472828 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.472982 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.473216 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.473287 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:19:58.473752 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.469717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.2.246.66:51063)
0927 15:19:58.470787 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.470812 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:58.470905 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:58.471468 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:58.471473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:58.471505 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:58.472415 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:19:58.472419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:58.472436 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:19:58.472491 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:58.473071 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:58.473075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:58.473080 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:19:58.473208 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 15:19:58.473295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":707,"threads_started":1}
I20260927 15:19:58.474062 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7ef6838afb47ee8cab32f163109e46" candidate_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f473ccc6944426a165a2081fd3b4a8" is_pre_election: true
I20260927 15:19:58.474725 8123 raft_consensus.cc:2454] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d111b0c0d941cbb0278f7bc04f1e54 in term 0.
I20260927 15:19:58.475351 7861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7ef6838afb47ee8cab32f163109e46" candidate_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac8c1953ad425799afb247c10e3bc3" is_pre_election: true
I20260927 15:19:58.475641 7926 leader_election.cc:302] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f473ccc6944426a165a2081fd3b4a8, b5d111b0c0d941cbb0278f7bc04f1e54; no voters:
I20260927 15:19:58.476090 8195 raft_consensus.cc:2781] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:58.476048 7861 raft_consensus.cc:2454] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d111b0c0d941cbb0278f7bc04f1e54 in term 0.
I20260927 15:19:58.476212 8195 raft_consensus.cc:482] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:58.476344 8195 raft_consensus.cc:3037] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:58.479043 8195 raft_consensus.cc:504] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.479828 8195 leader_election.cc:288] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [CANDIDATE]: Term 1 election: Requested vote from peers 98f473ccc6944426a165a2081fd3b4a8 (127.2.246.67:43507), 15ac8c1953ad425799afb247c10e3bc3 (127.2.246.65:39921)
I20260927 15:19:58.480556 7861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7ef6838afb47ee8cab32f163109e46" candidate_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac8c1953ad425799afb247c10e3bc3"
I20260927 15:19:58.480599 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7ef6838afb47ee8cab32f163109e46" candidate_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f473ccc6944426a165a2081fd3b4a8"
I20260927 15:19:58.480862 8123 raft_consensus.cc:3037] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:58.480854 7861 raft_consensus.cc:3037] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:58.483605 8123 raft_consensus.cc:2454] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d111b0c0d941cbb0278f7bc04f1e54 in term 1.
I20260927 15:19:58.484180 7926 leader_election.cc:302] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f473ccc6944426a165a2081fd3b4a8, b5d111b0c0d941cbb0278f7bc04f1e54; no voters:
I20260927 15:19:58.484544 7861 raft_consensus.cc:2454] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d111b0c0d941cbb0278f7bc04f1e54 in term 1.
I20260927 15:19:58.484619 8195 raft_consensus.cc:2781] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:58.485303 8195 raft_consensus.cc:686] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 1 LEADER]: Becoming Leader. State: Replica: b5d111b0c0d941cbb0278f7bc04f1e54, State: Running, Role: LEADER
I20260927 15:19:58.486308 8195 consensus_queue.cc:237] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } }
I20260927 15:19:58.492551 7719 catalog_manager.cc:5495] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5d111b0c0d941cbb0278f7bc04f1e54 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d111b0c0d941cbb0278f7bc04f1e54" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:58.518075 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.514757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.2.246.66:40255)
0927 15:19:58.515032 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.515041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.515147 (+ 106us) server_negotiation.cc:407] Connection header received
0927 15:19:58.515507 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.515512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.515617 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.515745 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.516819 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.516828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.516838 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.516856 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.516892 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.516906 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.517052 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.517580 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.517678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260927 15:19:58.523213 8169 tablet.cc:2369] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:58.529217 8123 raft_consensus.cc:1261] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 1 FOLLOWER]: Refusing update from remote peer b5d111b0c0d941cbb0278f7bc04f1e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:58.529654 7861 raft_consensus.cc:1261] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 1 FOLLOWER]: Refusing update from remote peer b5d111b0c0d941cbb0278f7bc04f1e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:58.530330 8198 consensus_queue.cc:1035] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f473ccc6944426a165a2081fd3b4a8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:58.530906 8195 consensus_queue.cc:1035] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ac8c1953ad425799afb247c10e3bc3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:58.547326 8203 mvcc.cc:205] Tried to move back new op lower bound from 7333979744363782144 to 7333979744205602816. Current Snapshot: MvccSnapshot[applied={T|T < 7333979744363782144}]
I20260927 15:19:58.561287 7719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:58.561651 7719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:58.566661 7719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41868a247f6422f9c466b48bd9fefb5: Sending DeleteTablet for 3 replicas of tablet 5f7ef6838afb47ee8cab32f163109e46
I20260927 15:19:58.568312 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:58 UTC) from {username='slave'} at 127.0.0.1:35172
I20260927 15:19:58.568243 7972 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:58 UTC) from {username='slave'} at 127.0.0.1:46648
I20260927 15:19:58.568388 7841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7ef6838afb47ee8cab32f163109e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:58 UTC) from {username='slave'} at 127.0.0.1:44174
I20260927 15:19:58.571591 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7779
I20260927 15:19:58.601419 8211 tablet_replica.cc:330] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: stopping tablet replica
I20260927 15:19:58.603665 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.574079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.2.246.65:39921)
0927 15:19:58.574360 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.574371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.598492 (+ 24121us) server_negotiation.cc:407] Connection header received
0927 15:19:58.598997 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.599008 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.599232 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.599342 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.602326 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.602338 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.602346 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.602383 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.602451 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.602475 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.602650 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.602952 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.603053 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:19:58.605624 7901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:58.606245 8211 raft_consensus.cc:2229] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:58.607414 8211 raft_consensus.cc:2258] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:58.608758 8212 tablet_replica.cc:330] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: stopping tablet replica
I20260927 15:19:58.609669 8212 raft_consensus.cc:2229] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:58.610426 8211 ts_tablet_manager.cc:1905] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:58.610548 8212 raft_consensus.cc:2258] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:58.615314 8212 ts_tablet_manager.cc:1905] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:58.621330 8211 ts_tablet_manager.cc:1918] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:58.621578 8211 log.cc:1196] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/5f7ef6838afb47ee8cab32f163109e46
I20260927 15:19:58.622373 8211 ts_tablet_manager.cc:1939] T 5f7ef6838afb47ee8cab32f163109e46 P b5d111b0c0d941cbb0278f7bc04f1e54: Deleting consensus metadata
I20260927 15:19:58.624449 7704 catalog_manager.cc:4841] TS b5d111b0c0d941cbb0278f7bc04f1e54 (127.2.246.66:40255): tablet 5f7ef6838afb47ee8cab32f163109e46 (table test-table [id=eeda04c2de69446a983a21af2514f3a0]) successfully deleted
I20260927 15:19:58.628896 8212 ts_tablet_manager.cc:1918] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:58.629137 8212 log.cc:1196] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/5f7ef6838afb47ee8cab32f163109e46
I20260927 15:19:58.629756 8212 ts_tablet_manager.cc:1939] T 5f7ef6838afb47ee8cab32f163109e46 P 98f473ccc6944426a165a2081fd3b4a8: Deleting consensus metadata
I20260927 15:19:58.632375 7705 catalog_manager.cc:4841] TS 98f473ccc6944426a165a2081fd3b4a8 (127.2.246.67:43507): tablet 5f7ef6838afb47ee8cab32f163109e46 (table test-table [id=eeda04c2de69446a983a21af2514f3a0]) successfully deleted
I20260927 15:19:58.717073 8213 tablet_replica.cc:330] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3: stopping tablet replica
I20260927 15:19:58.717641 8213 raft_consensus.cc:2229] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:58.718081 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7779
I20260927 15:19:58.718223 8213 raft_consensus.cc:2258] T 5f7ef6838afb47ee8cab32f163109e46 P 15ac8c1953ad425799afb247c10e3bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:19:58.736404 7705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:39921 (error 108)
I20260927 15:19:58.736783 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7910
I20260927 15:19:58.739048 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.738272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44186)
0927 15:19:58.738615 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.738810 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 15:19:58.740087 7705 catalog_manager.cc:4584] TS 15ac8c1953ad425799afb247c10e3bc3 (127.2.246.65:39921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f7ef6838afb47ee8cab32f163109e46: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
I20260927 15:19:58.741014 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.737941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.2.246.66:40255)
0927 15:19:58.738223 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.738232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.738263 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:19:58.738683 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.738688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.738761 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.738879 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.740105 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.740116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.740124 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.740149 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.740218 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.740239 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.740551 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.740630 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.740707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:19:58.741982 8032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:58.794176 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.793457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44190)
0927 15:19:58.793762 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.793937 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 15:19:58.795378 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.795007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44206)
0927 15:19:58.795168 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.795251 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:19:58.855124 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7910
I20260927 15:19:58.856381 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.855750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44222)
0927 15:19:58.856160 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.856239 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
I20260927 15:19:58.857599 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.857119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44232)
0927 15:19:58.857337 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.857423 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:19:58.874421 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8041
I20260927 15:19:58.878151 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.875429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.2.246.67:43507)
0927 15:19:58.875627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:58.875632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:58.875657 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:19:58.875816 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:58.875821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:58.875913 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:58.876034 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:58.877062 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:58.877068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:58.877072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:58.877097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:58.877145 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:58.877161 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:58.877426 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:58.877498 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 15:19:58.877597 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:19:58.878945 8163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:58.939213 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.938554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44238)
0927 15:19:58.938887 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.939025 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 15:19:58.940394 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:58.939994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:39921 (local address 127.0.0.1:44240)
0927 15:19:58.940183 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:58.940262 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 15:19:58.981738 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8041
I20260927 15:19:58.998916 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 7689
I20260927 15:19:59.000552 7750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:59.107720 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 7689
2026-09-27T15:19:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3186 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:19:59.151407 3033 mini_oidc.cc:152] Starting JWKS server
I20260927 15:19:59.155215 3033 webserver.cc:460] Webserver started at https://127.0.0.1:41249/ using document root <none> and password file <none>
I20260927 15:19:59.155532 3033 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:19:59.156235 3033 webserver.cc:460] Webserver started at http://127.0.0.1:38921/ using document root <none> and password file <none>
2026-09-27T15:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:59Z Disabled control of system clock
I20260927 15:19:59.186457 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34537
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:34537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41249/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/testchainca.cert with env {}
W20260927 15:19:59.534677 8225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:59.535004 8225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:59.535094 8225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:59.542979 8225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:59.543121 8225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:59.543162 8225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:59.543215 8225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:59.543253 8225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:19:59.543344 8225 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:19:59.543390 8225 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41249/jwks.json
I20260927 15:19:59.553802 8225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:34537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41249/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:19:59.555225 8225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:59.556492 8225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:59.570547 8231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:59.570626 8230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:59.571285 8233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:59.571841 8225 server_base.cc:1034] running on GCE node
I20260927 15:19:59.572902 8225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:59.574954 8225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:59.576386 8225 hybrid_clock.cc:648] HybridClock initialized: now 1790522399576313 us; error 76 us; skew 500 ppm
I20260927 15:19:59.576870 8225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:19:59.579908 8225 webserver.cc:460] Webserver started at http://127.2.246.126:44393/ using document root <none> and password file <none>
I20260927 15:19:59.580828 8225 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:59.580982 8225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:59.581365 8225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:59.585135 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "742d553939db4302a8d9cea7f5d30c61"
format_stamp: "Formatted at 2026-09-27 15:19:59 on dist-test-slave-14jq"
server_key: "e7c0c001ed0e7716ee2aa74cfb8c50b3"
server_key_iv: "017a514331c9d56ea2face24ace446d6"
server_key_version: "encryptionkey@0"
I20260927 15:19:59.586200 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "742d553939db4302a8d9cea7f5d30c61"
format_stamp: "Formatted at 2026-09-27 15:19:59 on dist-test-slave-14jq"
server_key: "e7c0c001ed0e7716ee2aa74cfb8c50b3"
server_key_iv: "017a514331c9d56ea2face24ace446d6"
server_key_version: "encryptionkey@0"
I20260927 15:19:59.593737 8225 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 15:19:59.598515 8239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:59.600958 8225 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:19:59.601243 8225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "742d553939db4302a8d9cea7f5d30c61"
format_stamp: "Formatted at 2026-09-27 15:19:59 on dist-test-slave-14jq"
server_key: "e7c0c001ed0e7716ee2aa74cfb8c50b3"
server_key_iv: "017a514331c9d56ea2face24ace446d6"
server_key_version: "encryptionkey@0"
I20260927 15:19:59.601549 8225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:59.619032 8225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:59.620348 8225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:59.620806 8225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:59.645493 8225 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:34537
I20260927 15:19:59.645517 8290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:34537 every 8 connection(s)
I20260927 15:19:59.647367 8225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:59.649276 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8225
I20260927 15:19:59.649506 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:19:59.650460 3033 external_mini_cluster.cc:1428] Setting key cdeaea2bc7245d3cc4008d66d1a67a99
I20260927 15:19:59.652938 8291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:59.667812 8291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61: Bootstrap starting.
I20260927 15:19:59.668515 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:59.652992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.2.246.126:34537)
0927 15:19:59.654792 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:59.654807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:59.654868 (+ 61us) server_negotiation.cc:407] Connection header received
0927 15:19:59.656700 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:59.656736 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:59.657398 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:59.658001 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:59.658682 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.660211 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.662299 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.663232 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.663572 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:59.665003 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:59.665013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:59.665034 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:19:59.665944 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:59.666065 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:19:59.666093 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:59.666209 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:59.666537 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:19:59.667062 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":875,"threads_started":1}
I20260927 15:19:59.671687 8291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:59.672819 8291 log.cc:824] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:59.674047 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:19:59.675836 8291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61: No bootstrap required, opened a new log
I20260927 15:19:59.683111 8291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } }
I20260927 15:19:59.683495 8291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:59.683627 8291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742d553939db4302a8d9cea7f5d30c61, State: Initialized, Role: FOLLOWER
I20260927 15:19:59.684252 8291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } }
I20260927 15:19:59.684492 8291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:59.684633 8291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:59.684829 8291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:59.687119 8291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } }
I20260927 15:19:59.687649 8291 leader_election.cc:302] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742d553939db4302a8d9cea7f5d30c61; no voters:
I20260927 15:19:59.688352 8291 leader_election.cc:288] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:59.688640 8296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:59.689512 8296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [term 1 LEADER]: Becoming Leader. State: Replica: 742d553939db4302a8d9cea7f5d30c61, State: Running, Role: LEADER
I20260927 15:19:59.690243 8296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } }
I20260927 15:19:59.690760 8291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:59.693418 8297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742d553939db4302a8d9cea7f5d30c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } } }
I20260927 15:19:59.693881 8297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:59.694955 8301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:59.694788 8298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742d553939db4302a8d9cea7f5d30c61. Latest consensus state: current_term: 1 leader_uuid: "742d553939db4302a8d9cea7f5d30c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d553939db4302a8d9cea7f5d30c61" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34537 } } }
I20260927 15:19:59.696266 8298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:59.702891 8301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:59.718192 8301 catalog_manager.cc:1347] Generated new cluster ID: 957f7bf7a09c4512a195dd40384eda71
I20260927 15:19:59.718420 8301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:59.734666 8301 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:59.736186 8301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:59.744652 8301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742d553939db4302a8d9cea7f5d30c61: Generated new TSK 0
I20260927 15:19:59.745677 8301 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:19:59.803799 8321 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:19:59.804006 8321 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:19:59.805406 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:59.774860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.2.246.126:34537)
0927 15:19:59.775059 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:59.775064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:59.775590 (+ 526us) server_negotiation.cc:407] Connection header received
0927 15:19:59.776011 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:59.776016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:59.776094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:59.776250 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:19:59.776828 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.777616 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.779127 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.779596 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.779733 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:59.780353 (+ 620us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:19:59.804550 (+ 24197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:19:59.805017 (+ 467us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 15:19:59.805625 8294 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:19:59.807331 8320 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:19:59.774658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:34537 (local address 127.0.0.1:35752)
0927 15:19:59.775452 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:59.775470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:59.775702 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:59.776273 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:59.776278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:59.776289 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:19:59.776643 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:59.776652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.777758 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.777763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:59.778953 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:59.778964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:59.779814 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:59.779821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:59.779964 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:59.780023 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:19:59.804920 (+ 24897us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:19:59.805327 (+ 407us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.246.126:34537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":414,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.246.126:34537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:19:59.809561 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8225
I20260927 15:19:59.811067 8286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:19:59.915757 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8225
2026-09-27T15:19:59Z chronyd exiting
I20260927 15:19:59.968415 3033 test_util.cc:175] -----------------------------------------------
I20260927 15:19:59.968621 3033 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522351366233-3033-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (822 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:19:59.972836 3033 mini_oidc.cc:152] Starting JWKS server
I20260927 15:19:59.975682 3033 webserver.cc:460] Webserver started at https://127.0.0.1:45713/ using document root <none> and password file <none>
I20260927 15:19:59.975952 3033 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:19:59.976728 3033 webserver.cc:460] Webserver started at http://127.0.0.1:33339/ using document root <none> and password file <none>
2026-09-27T15:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:59Z Disabled control of system clock
I20260927 15:20:00.013108 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:40105
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:40105
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45713/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/testchainca.cert with env {}
W20260927 15:20:00.362916 8328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:00.363245 8328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:00.363415 8328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:00.371093 8328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:00.371229 8328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:00.371266 8328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:00.371316 8328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:00.371362 8328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:20:00.371467 8328 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:20:00.371544 8328 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45713/jwks.json
I20260927 15:20:00.382073 8328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:40105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:40105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45713/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:00.383539 8328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:00.384883 8328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:00.398671 8334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:00.398689 8336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:00.398769 8333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:00.399648 8328 server_base.cc:1034] running on GCE node
I20260927 15:20:00.400588 8328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:00.402295 8328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:00.403599 8328 hybrid_clock.cc:648] HybridClock initialized: now 1790522400403538 us; error 52 us; skew 500 ppm
I20260927 15:20:00.404085 8328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:00.411041 8328 webserver.cc:460] Webserver started at http://127.2.246.126:32779/ using document root <none> and password file <none>
I20260927 15:20:00.411914 8328 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:00.412027 8328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:00.412392 8328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:00.415735 8328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "60563e74f9634f16b413b065dfd8ef53"
format_stamp: "Formatted at 2026-09-27 15:20:00 on dist-test-slave-14jq"
server_key: "4309faae73e01ee6d85c72d90847f867"
server_key_iv: "880b74bdb9120fe3813954dd6c2f7ac6"
server_key_version: "encryptionkey@0"
I20260927 15:20:00.416673 8328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "60563e74f9634f16b413b065dfd8ef53"
format_stamp: "Formatted at 2026-09-27 15:20:00 on dist-test-slave-14jq"
server_key: "4309faae73e01ee6d85c72d90847f867"
server_key_iv: "880b74bdb9120fe3813954dd6c2f7ac6"
server_key_version: "encryptionkey@0"
I20260927 15:20:00.423173 8328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 15:20:00.428151 8342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:00.430048 8328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:20:00.430298 8328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "60563e74f9634f16b413b065dfd8ef53"
format_stamp: "Formatted at 2026-09-27 15:20:00 on dist-test-slave-14jq"
server_key: "4309faae73e01ee6d85c72d90847f867"
server_key_iv: "880b74bdb9120fe3813954dd6c2f7ac6"
server_key_version: "encryptionkey@0"
I20260927 15:20:00.430608 8328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:00.457151 8328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:00.458190 8328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:00.458570 8328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:00.481259 8328 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:40105
I20260927 15:20:00.481323 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:40105 every 8 connection(s)
I20260927 15:20:00.482839 8328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:00.487857 8394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:00.492233 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8328
I20260927 15:20:00.492473 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:00.493247 3033 external_mini_cluster.cc:1428] Setting key 6923d08459ca34ccf27658f3226dd24d
I20260927 15:20:00.503984 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53: Bootstrap starting.
I20260927 15:20:00.508919 8394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:00.510388 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:00.495797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.2.246.126:40105)
0927 15:20:00.496973 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:00.496985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:00.497033 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:20:00.499051 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:00.499078 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:00.499687 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:00.500140 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:00.501022 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.502718 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.504764 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.505670 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.505909 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:00.507458 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:00.507466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:00.507485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:00.508133 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:00.508238 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:00.508261 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:00.508433 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:00.508690 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 15:20:00.509089 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":433,"threads_started":1}
I20260927 15:20:00.510520 8394 log.cc:824] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:00.514642 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53: No bootstrap required, opened a new log
I20260927 15:20:00.515360 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:00.524048 8394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } }
I20260927 15:20:00.524781 8394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:00.524923 8394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60563e74f9634f16b413b065dfd8ef53, State: Initialized, Role: FOLLOWER
I20260927 15:20:00.525738 8394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } }
I20260927 15:20:00.526126 8394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:00.526305 8394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:00.526541 8394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:00.529597 8394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } }
I20260927 15:20:00.530402 8394 leader_election.cc:302] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60563e74f9634f16b413b065dfd8ef53; no voters:
I20260927 15:20:00.531387 8394 leader_election.cc:288] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:00.531596 8399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:00.532687 8399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [term 1 LEADER]: Becoming Leader. State: Replica: 60563e74f9634f16b413b065dfd8ef53, State: Running, Role: LEADER
I20260927 15:20:00.533391 8399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } }
I20260927 15:20:00.534462 8394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:00.537384 8401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60563e74f9634f16b413b065dfd8ef53. Latest consensus state: current_term: 1 leader_uuid: "60563e74f9634f16b413b065dfd8ef53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } } }
I20260927 15:20:00.537302 8400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60563e74f9634f16b413b065dfd8ef53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60563e74f9634f16b413b065dfd8ef53" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40105 } } }
I20260927 15:20:00.538020 8401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:00.538058 8400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:00.539773 8405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:00.545727 8405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:00.556921 8405 catalog_manager.cc:1347] Generated new cluster ID: f51032fd1c34473ea7039be3a7a65e1c
I20260927 15:20:00.557611 8405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:00.576009 8405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:00.577685 8405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:00.584883 8405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60563e74f9634f16b413b065dfd8ef53: Generated new TSK 0
I20260927 15:20:00.585773 8405 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:20:00.634282 8424 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:20:00.634479 8424 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:20:00.635905 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:00.607148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.2.246.126:40105)
0927 15:20:00.607422 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:00.607434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:00.607733 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:20:00.608046 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:00.608050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:00.608143 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:00.608354 (+ 211us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:20:00.608832 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.609654 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.611278 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.611833 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.611993 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:00.612535 (+ 542us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:20:00.635051 (+ 22516us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:20:00.635529 (+ 478us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 15:20:00.636063 8397 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 15:20:00.636500 8423 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:00.606942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:40105 (local address 127.0.0.1:60142)
0927 15:20:00.607595 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:00.607618 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:00.607819 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:00.608356 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:00.608361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:00.608373 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:20:00.608669 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:00.608678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.609865 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.609872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:00.611106 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:00.611117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:00.612025 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:00.612032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:00.612202 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:00.612240 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:20:00.635385 (+ 23145us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:20:00.635818 (+ 433us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.246.126:40105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":506,"threads_started":1}
I20260927 15:20:00.637742 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8328
I20260927 15:20:00.639681 8389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:00.747648 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8328
2026-09-27T15:20:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (834 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:20:00.807953 3033 mini_oidc.cc:152] Starting JWKS server
I20260927 15:20:00.810222 3033 webserver.cc:460] Webserver started at https://127.0.0.1:43573/ using document root <none> and password file <none>
I20260927 15:20:00.810534 3033 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:20:00.811292 3033 webserver.cc:460] Webserver started at http://127.0.0.1:45345/ using document root <none> and password file <none>
2026-09-27T15:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:00Z Disabled control of system clock
I20260927 15:20:00.846395 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44475
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43573/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:20:01.189561 8431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:01.190003 8431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:01.190168 8431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:01.197822 8431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:01.198051 8431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:01.198138 8431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:01.198248 8431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:01.198335 8431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:20:01.198549 8431 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:20:01.198632 8431 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43573/jwks.json
I20260927 15:20:01.208389 8431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43573/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:01.209906 8431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:01.211863 8431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:01.225139 8439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:01.225219 8437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:01.225044 8436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:01.226944 8431 server_base.cc:1034] running on GCE node
I20260927 15:20:01.227948 8431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:01.229804 8431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:01.231261 8431 hybrid_clock.cc:648] HybridClock initialized: now 1790522401231198 us; error 61 us; skew 500 ppm
I20260927 15:20:01.231730 8431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:01.234659 8431 webserver.cc:460] Webserver started at http://127.2.246.126:45051/ using document root <none> and password file <none>
I20260927 15:20:01.235486 8431 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:01.235627 8431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:01.235975 8431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:01.239063 8431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "12ef82b50d19479d87116460e0c527fc"
format_stamp: "Formatted at 2026-09-27 15:20:01 on dist-test-slave-14jq"
server_key: "ba3f19812ce05ed5766b4bfc22e12f79"
server_key_iv: "33321a52bc0fae1f5aee94a060fb5de7"
server_key_version: "encryptionkey@0"
I20260927 15:20:01.239898 8431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "12ef82b50d19479d87116460e0c527fc"
format_stamp: "Formatted at 2026-09-27 15:20:01 on dist-test-slave-14jq"
server_key: "ba3f19812ce05ed5766b4bfc22e12f79"
server_key_iv: "33321a52bc0fae1f5aee94a060fb5de7"
server_key_version: "encryptionkey@0"
I20260927 15:20:01.246348 8431 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:20:01.251353 8445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:01.253259 8431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:20:01.253541 8431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "12ef82b50d19479d87116460e0c527fc"
format_stamp: "Formatted at 2026-09-27 15:20:01 on dist-test-slave-14jq"
server_key: "ba3f19812ce05ed5766b4bfc22e12f79"
server_key_iv: "33321a52bc0fae1f5aee94a060fb5de7"
server_key_version: "encryptionkey@0"
I20260927 15:20:01.253821 8431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:01.272392 8431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:01.273342 8431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:01.273703 8431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:01.296540 8431 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44475
I20260927 15:20:01.296644 8496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44475 every 8 connection(s)
I20260927 15:20:01.298056 8431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:01.302894 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8431
I20260927 15:20:01.303172 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:01.303692 8497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:01.304123 3033 external_mini_cluster.cc:1428] Setting key 901533ab06ca74ff5c4161d608cb0553
I20260927 15:20:01.322238 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:01.306853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.2.246.126:44475)
0927 15:20:01.308190 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:01.308204 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:01.308266 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:20:01.310184 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:01.310210 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:01.310789 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:01.311294 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:01.312447 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.313966 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.316286 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.317121 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.317288 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:01.319191 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:01.319199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:01.319220 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:01.319937 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:01.320046 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:01.320070 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:01.320181 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:01.320578 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 15:20:01.321035 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":508,"threads_started":1}
I20260927 15:20:01.324256 8497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc: Bootstrap starting.
I20260927 15:20:01.327958 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:01.329466 8497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:01.332086 8497 log.cc:824] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:01.336745 8497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc: No bootstrap required, opened a new log
I20260927 15:20:01.346585 8497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } }
I20260927 15:20:01.347149 8497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:01.347270 8497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ef82b50d19479d87116460e0c527fc, State: Initialized, Role: FOLLOWER
I20260927 15:20:01.348059 8497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } }
I20260927 15:20:01.348361 8497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:01.348529 8497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:01.348737 8497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:01.351497 8497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } }
I20260927 15:20:01.352574 8497 leader_election.cc:302] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ef82b50d19479d87116460e0c527fc; no voters:
I20260927 15:20:01.353396 8497 leader_election.cc:288] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:01.353657 8502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:01.354903 8502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [term 1 LEADER]: Becoming Leader. State: Replica: 12ef82b50d19479d87116460e0c527fc, State: Running, Role: LEADER
I20260927 15:20:01.355592 8502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } }
I20260927 15:20:01.356424 8497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:01.359864 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ef82b50d19479d87116460e0c527fc. Latest consensus state: current_term: 1 leader_uuid: "12ef82b50d19479d87116460e0c527fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } } }
I20260927 15:20:01.360524 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:01.360275 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ef82b50d19479d87116460e0c527fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ef82b50d19479d87116460e0c527fc" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44475 } } }
I20260927 15:20:01.360949 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:01.363792 8508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:01.375231 8508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:01.391561 8508 catalog_manager.cc:1347] Generated new cluster ID: c74ce5588c9f49aca5e454cfda0ed8fe
I20260927 15:20:01.391754 8508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:01.406385 8508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:01.408033 8508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:01.420465 8508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ef82b50d19479d87116460e0c527fc: Generated new TSK 0
I20260927 15:20:01.421766 8508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:01.453423 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:01.442284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.2.246.126:44475)
0927 15:20:01.442600 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:01.442610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:01.443451 (+ 841us) server_negotiation.cc:407] Connection header received
0927 15:20:01.443756 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:01.443761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:01.443892 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:01.444073 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:20:01.444753 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.445598 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.448124 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.448759 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.448916 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:01.449768 (+ 852us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:20:01.452645 (+ 2877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43573/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:20:01.453110 (+ 465us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38354: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43573/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":97}
W20260927 15:20:01.453630 8500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38354: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43573/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 15:20:01.457412 8526 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:01.442262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44475 (local address 127.0.0.1:38354)
0927 15:20:01.443115 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:01.443141 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:01.443377 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:01.444149 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:01.444156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:01.444174 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:20:01.444540 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:01.444554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.445796 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.445803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:01.447839 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:01.447860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:01.449054 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:01.449064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:01.449277 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:01.449330 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:20:01.452976 (+ 3646us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43573/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:20:01.456819 (+ 3843us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.246.126:44475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43573/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":3364,"thread_start_us":437,"threads_started":1}
I20260927 15:20:01.459085 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8431
I20260927 15:20:01.460908 8492 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 15:20:01.469051 8527 webserver.cc:533] Webserver: sslize failed:
E20260927 15:20:01.469229 8527 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:20:01.558511 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8431
2026-09-27T15:20:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (818 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:01Z Disabled control of system clock
I20260927 15:20:01.659186 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35467
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35467
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/testchainca.cert with env {}
W20260927 15:20:02.016032 8532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:02.016386 8532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:02.016482 8532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:02.025043 8532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:02.025203 8532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:02.025243 8532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:02.025296 8532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:02.025333 8532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:20:02.025398 8532 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:20:02.025430 8532 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:20:02.036360 8532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:02.037753 8532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:02.039153 8532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:02.057175 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:02.057771 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:02.057920 8532 server_base.cc:1034] running on GCE node
W20260927 15:20:02.058288 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:02.059351 8532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:02.061132 8532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:02.064668 8532 hybrid_clock.cc:648] HybridClock initialized: now 1790522402064557 us; error 94 us; skew 500 ppm
I20260927 15:20:02.065222 8532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:02.068881 8532 webserver.cc:460] Webserver started at http://127.2.246.126:44665/ using document root <none> and password file <none>
I20260927 15:20:02.069803 8532 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:02.069993 8532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:02.070385 8532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:02.073671 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "1cbf818fa59a489694217342e297f375"
format_stamp: "Formatted at 2026-09-27 15:20:02 on dist-test-slave-14jq"
server_key: "b247cf54ce3940335c24bdc9d6b54628"
server_key_iv: "8558554bf897f69f4c57b12950bdd8e9"
server_key_version: "encryptionkey@0"
I20260927 15:20:02.074714 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "1cbf818fa59a489694217342e297f375"
format_stamp: "Formatted at 2026-09-27 15:20:02 on dist-test-slave-14jq"
server_key: "b247cf54ce3940335c24bdc9d6b54628"
server_key_iv: "8558554bf897f69f4c57b12950bdd8e9"
server_key_version: "encryptionkey@0"
I20260927 15:20:02.082008 8532 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 15:20:02.087812 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:02.090413 8532 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:20:02.090696 8532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "1cbf818fa59a489694217342e297f375"
format_stamp: "Formatted at 2026-09-27 15:20:02 on dist-test-slave-14jq"
server_key: "b247cf54ce3940335c24bdc9d6b54628"
server_key_iv: "8558554bf897f69f4c57b12950bdd8e9"
server_key_version: "encryptionkey@0"
I20260927 15:20:02.090975 8532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:02.106405 8532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:02.107366 8532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:02.107679 8532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:02.131438 8532 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:35467
I20260927 15:20:02.131588 8597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:35467 every 8 connection(s)
I20260927 15:20:02.132998 8532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:02.134294 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8532
I20260927 15:20:02.134501 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:02.135177 3033 external_mini_cluster.cc:1428] Setting key 986de57ee4136a19760e97e3fc9f6c02
I20260927 15:20:02.139727 8598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:02.160214 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:02.137592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.2.246.126:35467)
0927 15:20:02.139054 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:02.139076 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:02.139152 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:20:02.142357 (+ 3205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:02.142400 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:02.143557 (+ 1157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:02.144250 (+ 693us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:02.145136 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.148652 (+ 3516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.150695 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.153311 (+ 2616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.153668 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:02.155603 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:02.155617 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:02.155641 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:02.156454 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:02.156616 (+ 162us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:02.156653 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:02.156821 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:02.157425 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 15:20:02.158068 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":472,"threads_started":1}
I20260927 15:20:02.162613 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375: Bootstrap starting.
I20260927 15:20:02.167951 8598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:02.168013 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:02.169473 8598 log.cc:824] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:02.173791 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375: No bootstrap required, opened a new log
I20260927 15:20:02.182523 8598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } }
I20260927 15:20:02.183063 8598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:02.183173 8598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbf818fa59a489694217342e297f375, State: Initialized, Role: FOLLOWER
I20260927 15:20:02.183938 8598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } }
I20260927 15:20:02.184243 8598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:02.184430 8598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:02.184649 8598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:02.187575 8598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } }
I20260927 15:20:02.188366 8598 leader_election.cc:302] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cbf818fa59a489694217342e297f375; no voters:
I20260927 15:20:02.189208 8598 leader_election.cc:288] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:02.189491 8603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:02.191640 8603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbf818fa59a489694217342e297f375, State: Running, Role: LEADER
I20260927 15:20:02.192663 8598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:02.192517 8603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } }
I20260927 15:20:02.198963 8605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cbf818fa59a489694217342e297f375. Latest consensus state: current_term: 1 leader_uuid: "1cbf818fa59a489694217342e297f375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } } }
I20260927 15:20:02.199714 8605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:02.198992 8604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cbf818fa59a489694217342e297f375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf818fa59a489694217342e297f375" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35467 } } }
I20260927 15:20:02.200071 8604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:02.201256 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:02.209196 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:02.222688 8614 catalog_manager.cc:1347] Generated new cluster ID: 23be74b1fce641b2b1bf8d2a42a68d2e
I20260927 15:20:02.222916 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:02.247020 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:02.248497 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:02.258837 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cbf818fa59a489694217342e297f375: Generated new TSK 0
I20260927 15:20:02.259884 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:02.362766 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:02.285555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.2.246.126:35467)
0927 15:20:02.285817 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:02.285823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:02.286499 (+ 676us) server_negotiation.cc:407] Connection header received
0927 15:20:02.286868 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:02.286873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:02.286981 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:02.287178 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:20:02.288486 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.289299 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.292421 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.293094 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.293237 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:02.294170 (+ 933us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:20:02.362177 (+ 68007us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:20:02.362530 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 15:20:02.362954 8601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:20:02.363850 8627 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:02.285156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35467 (local address 127.0.0.1:50260)
0927 15:20:02.286286 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:02.286316 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:02.286586 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:02.287238 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:02.287244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:02.287274 (+ 30us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:20:02.288110 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:02.288143 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.289484 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.289491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:02.291973 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:02.292008 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:02.293360 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:02.293369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:02.293651 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:02.293731 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:20:02.362610 (+ 68879us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:20:02.363102 (+ 492us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.246.126:35467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":559,"threads_started":1}
I20260927 15:20:02.365723 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8532
I20260927 15:20:02.367594 8593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:02.472294 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8532
2026-09-27T15:20:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (888 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (538 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (486 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:03 dist-test-slave-14jq krb5kdc[8642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:03 dist-test-slave-14jq krb5kdc[8642](info): set up 2 sockets
Sep 27 15:20:03 dist-test-slave-14jq krb5kdc[8642](info): commencing operation
krb5kdc: starting...
W20260927 15:20:04.655527 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:20:04 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:20:04.919896 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37825
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:05.287241 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:05.287585 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:05.287678 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:05.295352 8655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:05.295486 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:05.295532 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:05.295583 8655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:05.295619 8655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:05.306432 8655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:05.307768 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:05.309026 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:05.323390 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:05.323300 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:05.323225 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:05.324551 8655 server_base.cc:1034] running on GCE node
I20260927 15:20:05.325675 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:05.327631 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:05.328895 8655 hybrid_clock.cc:648] HybridClock initialized: now 1790522405328831 us; error 45 us; skew 500 ppm
I20260927 15:20:05.329389 8655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:05 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:05.333982 8655 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:20:05.336308 8655 webserver.cc:460] Webserver started at http://127.2.246.126:41521/ using document root <none> and password file <none>
I20260927 15:20:05.337718 8655 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:05.338001 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:05.338495 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:05.343848 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "f51e3539413044939bfb802dc2c383ae"
format_stamp: "Formatted at 2026-09-27 15:20:05 on dist-test-slave-14jq"
server_key: "2f9ddf7dccb69ba96ef024c186a0c3e5"
server_key_iv: "fc9e6c7b101c09ef0f869b341eeb3a95"
server_key_version: "encryptionkey@0"
I20260927 15:20:05.345568 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "f51e3539413044939bfb802dc2c383ae"
format_stamp: "Formatted at 2026-09-27 15:20:05 on dist-test-slave-14jq"
server_key: "2f9ddf7dccb69ba96ef024c186a0c3e5"
server_key_iv: "fc9e6c7b101c09ef0f869b341eeb3a95"
server_key_version: "encryptionkey@0"
I20260927 15:20:05.353070 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260927 15:20:05.359619 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:05.361512 8655 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260927 15:20:05.361712 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "f51e3539413044939bfb802dc2c383ae"
format_stamp: "Formatted at 2026-09-27 15:20:05 on dist-test-slave-14jq"
server_key: "2f9ddf7dccb69ba96ef024c186a0c3e5"
server_key_iv: "fc9e6c7b101c09ef0f869b341eeb3a95"
server_key_version: "encryptionkey@0"
I20260927 15:20:05.362030 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:05.381673 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:05.387379 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:05.387698 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:05.411966 8655 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:37825
I20260927 15:20:05.412112 8721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:37825 every 8 connection(s)
I20260927 15:20:05.413640 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:05.417119 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8655
I20260927 15:20:05.417405 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:05.418418 3033 external_mini_cluster.cc:1428] Setting key 05b7f557e69cb18344da0eebac8ae9cf
I20260927 15:20:05.421598 8722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:05.439987 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae: Bootstrap starting.
Sep 27 15:20:05 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:05.445552 8722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:05.447625 8722 log.cc:824] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:05.452970 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae: No bootstrap required, opened a new log
I20260927 15:20:05.457557 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:05.421397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.2.246.126:37825)
0927 15:20:05.423035 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:05.423057 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:05.423126 (+ 69us) server_negotiation.cc:407] Connection header received
0927 15:20:05.425918 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:05.425963 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:05.426811 (+ 848us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:05.427493 (+ 682us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:05.429095 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:05.431455 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:05.433019 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:05.434309 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:05.434602 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:05.445424 (+ 10822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:05.445444 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:05.445476 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:05.445565 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:05.450745 (+ 5180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:05.451900 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:05.451913 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:05.451924 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:05.452225 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:05.452653 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:05.452658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:05.452662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:05.453514 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:05.453964 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:05.454560 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 15:20:05.455421 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":705,"threads_started":1}
I20260927 15:20:05.465258 8722 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } }
I20260927 15:20:05.465971 8722 raft_consensus.cc:374] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:05.466120 8722 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51e3539413044939bfb802dc2c383ae, State: Initialized, Role: FOLLOWER
I20260927 15:20:05.467031 8722 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } }
I20260927 15:20:05.467554 8722 raft_consensus.cc:388] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:05.467764 8722 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:05.468026 8722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:05.471493 8722 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } }
I20260927 15:20:05.472553 8722 leader_election.cc:302] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51e3539413044939bfb802dc2c383ae; no voters:
I20260927 15:20:05.473495 8722 leader_election.cc:288] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:05.473654 8727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:05.474666 8727 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [term 1 LEADER]: Becoming Leader. State: Replica: f51e3539413044939bfb802dc2c383ae, State: Running, Role: LEADER
I20260927 15:20:05.475319 8727 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } }
I20260927 15:20:05.476454 8722 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:05.478365 8728 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51e3539413044939bfb802dc2c383ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } } }
I20260927 15:20:05.478749 8728 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:05.479559 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51e3539413044939bfb802dc2c383ae. Latest consensus state: current_term: 1 leader_uuid: "f51e3539413044939bfb802dc2c383ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51e3539413044939bfb802dc2c383ae" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37825 } } }
I20260927 15:20:05.479827 8732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:05.479892 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:05.489533 8732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:05.503501 8732 catalog_manager.cc:1347] Generated new cluster ID: acc652851a8a43db905ad9f1424db2df
I20260927 15:20:05.503646 8732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:05.522451 8732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:05.524300 8732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:05.538967 8732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51e3539413044939bfb802dc2c383ae: Generated new TSK 0
I20260927 15:20:05.540452 8732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:20:05.611418 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:06.067595 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:06.068002 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:06.068195 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:06.077665 8748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:06.077915 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:06.078161 8748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:06.090530 8748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:06.092427 8748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:06.094213 8748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:06.112003 8754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:06.112296 8756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:06.112628 8753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:06.114202 8748 server_base.cc:1034] running on GCE node
I20260927 15:20:06.115080 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:06.116379 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:06.117714 8748 hybrid_clock.cc:648] HybridClock initialized: now 1790522406117628 us; error 70 us; skew 500 ppm
I20260927 15:20:06.118245 8748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:06 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:06.122464 8748 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:20:06.124686 8748 webserver.cc:460] Webserver started at http://127.2.246.65:35561/ using document root <none> and password file <none>
I20260927 15:20:06.125702 8748 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:06.125907 8748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:06.126324 8748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:06.130656 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "751d6359a51d4499ac0bc0bf152bbdcf"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "492a809832a90d32fc59fb72bcc078ee"
server_key_iv: "02a4731b608dae6439667f5c19a1d3f8"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.131822 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "751d6359a51d4499ac0bc0bf152bbdcf"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "492a809832a90d32fc59fb72bcc078ee"
server_key_iv: "02a4731b608dae6439667f5c19a1d3f8"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.138842 8748 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 15:20:06.143899 8763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.146116 8748 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:06.146415 8748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "751d6359a51d4499ac0bc0bf152bbdcf"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "492a809832a90d32fc59fb72bcc078ee"
server_key_iv: "02a4731b608dae6439667f5c19a1d3f8"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.146734 8748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:06.168671 8748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:06.175251 8748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:06.175702 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:06.177122 8748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:06.179062 8748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:06.179183 8748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.179366 8748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:06.179442 8748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.222936 8748 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45291
I20260927 15:20:06.222994 8875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45291 every 8 connection(s)
I20260927 15:20:06.224709 8748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:06.232659 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8748
I20260927 15:20:06.233060 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:06.234165 3033 external_mini_cluster.cc:1428] Setting key 6300aab218832718d673d15896ea52c4
Sep 27 15:20:06 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:06.258018 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:06.228364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:46765 (local address 127.2.246.126:37825)
0927 15:20:06.228907 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:06.228911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:06.230624 (+ 1713us) server_negotiation.cc:407] Connection header received
0927 15:20:06.232705 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:06.232711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:06.232818 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:06.232950 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:06.235998 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.237073 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.240621 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.241148 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.241297 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:06.248181 (+ 6884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:06.248192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:06.248200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:06.248271 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:06.252797 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:06.254144 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:06.254155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:06.254162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:06.254386 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:06.255071 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:06.255079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:06.255085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:06.255554 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:06.255916 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:06.256975 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 15:20:06.257343 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260927 15:20:06.259152 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:06.229057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37825 (local address 127.2.246.65:46765)
0927 15:20:06.230404 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:06.230463 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:06.232353 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:06.233400 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:06.233479 (+ 79us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:06.234789 (+ 1310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:06.235742 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:06.235772 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.237337 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.237343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:06.240290 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:06.240332 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.241538 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.241550 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:06.242056 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:06.243201 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:06.243254 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:06.247627 (+ 4373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:06.253134 (+ 5507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:06.253146 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:06.253164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:06.253828 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:06.254601 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:06.254605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:06.254608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:06.254899 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:06.255829 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:06.255840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:06.256376 (+ 536us) client_negotiation.cc:769] Sending connection context
0927 15:20:06.256700 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 15:20:06.257309 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":573,"threads_started":1}
I20260927 15:20:06.261746 8876 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37825
I20260927 15:20:06.262319 8876 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:06.263397 8876 heartbeater.cc:511] Master 127.2.246.126:37825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:20:06.266651 8687 ts_manager.cc:194] Registered new tserver with Master: 751d6359a51d4499ac0bc0bf152bbdcf (127.2.246.65:45291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:20:06.268590 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:46765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:20:06.298316 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:06.667580 8882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:06.667905 8882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:06.668001 8882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:06.675724 8882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:06.675850 8882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:06.676033 8882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:06.686606 8882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:06.688046 8882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:06.689473 8882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:06.705602 8887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:06.705602 8888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:06.705894 8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:06.707141 8882 server_base.cc:1034] running on GCE node
I20260927 15:20:06.708029 8882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:06.709595 8882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:06.710886 8882 hybrid_clock.cc:648] HybridClock initialized: now 1790522406710819 us; error 107 us; skew 500 ppm
I20260927 15:20:06.711385 8882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:06 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:06.715727 8882 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:20:06.717900 8882 webserver.cc:460] Webserver started at http://127.2.246.66:40407/ using document root <none> and password file <none>
I20260927 15:20:06.718856 8882 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:06.719033 8882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:06.719417 8882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:06.723949 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "f8c352baf3544b1da54956b2b7a8b58d"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "42af5ffa27068ae9ed8a1372b22714b9"
server_key_iv: "708e644b8ebc3fe04bae20eedd4b5424"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.725113 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "f8c352baf3544b1da54956b2b7a8b58d"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "42af5ffa27068ae9ed8a1372b22714b9"
server_key_iv: "708e644b8ebc3fe04bae20eedd4b5424"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.733091 8882 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20260927 15:20:06.738922 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.741516 8882 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:06.741886 8882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "f8c352baf3544b1da54956b2b7a8b58d"
format_stamp: "Formatted at 2026-09-27 15:20:06 on dist-test-slave-14jq"
server_key: "42af5ffa27068ae9ed8a1372b22714b9"
server_key_iv: "708e644b8ebc3fe04bae20eedd4b5424"
server_key_version: "encryptionkey@0"
I20260927 15:20:06.742211 8882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:06.760506 8882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:06.767210 8882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:06.767849 8882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:06.769243 8882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:06.771520 8882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:06.771687 8882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.771862 8882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:06.771945 8882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:06.815187 8882 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:43765
I20260927 15:20:06.815239 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:43765 every 8 connection(s)
I20260927 15:20:06.816908 8882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:06.822717 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 8882
I20260927 15:20:06.822925 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:06.823513 3033 external_mini_cluster.cc:1428] Setting key 688575d00d2ca0c3c7a03958980d3e93
Sep 27 15:20:06 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:06.846421 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:06.820387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:35929 (local address 127.2.246.126:37825)
0927 15:20:06.820951 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:06.820957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:06.822573 (+ 1616us) server_negotiation.cc:407] Connection header received
0927 15:20:06.825239 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:06.825246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:06.825347 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:06.825521 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:06.827959 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.828780 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.831722 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.832270 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.832434 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:06.838003 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:06.838016 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:06.838023 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:06.838087 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:06.841000 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:06.842917 (+ 1917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:06.842923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:06.842925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:06.843110 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:06.843859 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:06.843863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:06.843866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:06.844228 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:06.844364 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:06.845795 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0927 15:20:06.846118 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":288,"threads_started":1}
I20260927 15:20:06.848624 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:06.821113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37825 (local address 127.2.246.66:35929)
0927 15:20:06.822201 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:06.822307 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:06.824792 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:06.825785 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:06.825794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:06.826706 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:06.827664 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:06.827696 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.829006 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.829014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:06.831393 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:06.831416 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:06.832624 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:06.832634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:06.833013 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:06.834035 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:06.834062 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:06.837417 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:06.841549 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:06.841572 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:06.841604 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:06.842616 (+ 1012us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:06.843267 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:06.843275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:06.843282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:06.843670 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:06.844399 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:06.844413 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:06.845093 (+ 680us) client_negotiation.cc:769] Sending connection context
0927 15:20:06.845504 (+ 411us) client_negotiation.cc:240] Negotiation successful
0927 15:20:06.846406 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:20:06.850858 9010 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37825
I20260927 15:20:06.851259 9010 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:06.852352 9010 heartbeater.cc:511] Master 127.2.246.126:37825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:20:06.854609 8687 ts_manager.cc:194] Registered new tserver with Master: f8c352baf3544b1da54956b2b7a8b58d (127.2.246.66:43765)
I20260927 15:20:06.855666 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:35929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:20:06.886353 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:07.273411 8876 heartbeater.cc:503] Master 127.2.246.126:37825 was elected leader, sending a full tablet report...
W20260927 15:20:07.318650 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:07.319015 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:07.319161 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:07.326583 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:07.326716 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:07.326836 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:07.336752 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:07.338148 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:07.339504 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:07.356367 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:07.356367 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:07.356317 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:07.357254 9016 server_base.cc:1034] running on GCE node
I20260927 15:20:07.358166 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:07.359591 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:07.360991 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790522407360877 us; error 94 us; skew 500 ppm
I20260927 15:20:07.361549 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.366138 9016 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:20:07.368317 9016 webserver.cc:460] Webserver started at http://127.2.246.67:34907/ using document root <none> and password file <none>
I20260927 15:20:07.369181 9016 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:07.369349 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:07.369704 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:07.373088 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "917b0bcef4134952959fd6fe59a3a1f3"
format_stamp: "Formatted at 2026-09-27 15:20:07 on dist-test-slave-14jq"
server_key: "91e5aab0f1ff88b49569b828d88403c9"
server_key_iv: "0b8798fbc0478581cafb3fa2ee745636"
server_key_version: "encryptionkey@0"
I20260927 15:20:07.374101 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "917b0bcef4134952959fd6fe59a3a1f3"
format_stamp: "Formatted at 2026-09-27 15:20:07 on dist-test-slave-14jq"
server_key: "91e5aab0f1ff88b49569b828d88403c9"
server_key_iv: "0b8798fbc0478581cafb3fa2ee745636"
server_key_version: "encryptionkey@0"
I20260927 15:20:07.380805 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:20:07.386422 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:07.388549 9016 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:20:07.388815 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "917b0bcef4134952959fd6fe59a3a1f3"
format_stamp: "Formatted at 2026-09-27 15:20:07 on dist-test-slave-14jq"
server_key: "91e5aab0f1ff88b49569b828d88403c9"
server_key_iv: "0b8798fbc0478581cafb3fa2ee745636"
server_key_version: "encryptionkey@0"
I20260927 15:20:07.389099 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:07.407138 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:07.412935 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:07.413267 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:07.414603 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:07.416498 9016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:07.416608 9016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:07.416723 9016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:07.416787 9016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:07.458901 9016 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:34489
I20260927 15:20:07.458936 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:34489 every 8 connection(s)
I20260927 15:20:07.460287 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:07.464469 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9016
I20260927 15:20:07.464674 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:07.465325 3033 external_mini_cluster.cc:1428] Setting key bbcf809adbd5a29ebf439202f2ae29e3
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:07.488997 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.463946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:33205 (local address 127.2.246.126:37825)
0927 15:20:07.464590 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.464600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.466148 (+ 1548us) server_negotiation.cc:407] Connection header received
0927 15:20:07.469495 (+ 3347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.469502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.469643 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.470098 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.472770 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.473646 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.475914 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.476495 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.476678 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.481705 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.481712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.481716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.481757 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.485045 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.486261 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.486266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.486269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.486400 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.487027 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.487031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.487033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.487318 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.487454 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.488514 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.488698 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":318,"threads_started":1}
I20260927 15:20:07.490417 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.464634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37825 (local address 127.2.246.67:33205)
0927 15:20:07.465879 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:07.465980 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:07.468935 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:07.470439 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:07.470450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:07.471273 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:07.472449 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:07.472479 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.473900 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.473906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:07.475457 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:07.475475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.476881 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.476890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:07.477208 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.478315 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:07.478345 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:07.481361 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:07.485288 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:07.485300 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:07.485321 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:07.486097 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:07.486567 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:07.486572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:07.486574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:07.486855 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:07.487477 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:07.487485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:07.488016 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 15:20:07.488328 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 15:20:07.488877 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":415,"threads_started":1}
I20260927 15:20:07.492257 9144 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37825
I20260927 15:20:07.492725 9144 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:07.494119 9144 heartbeater.cc:511] Master 127.2.246.126:37825 requested a full tablet report, sending...
I20260927 15:20:07.496076 8687 ts_manager.cc:194] Registered new tserver with Master: 917b0bcef4134952959fd6fe59a3a1f3 (127.2.246.67:34489)
I20260927 15:20:07.497033 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:33205
I20260927 15:20:07.509033 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/test-admin.keytab.
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.541139 3033 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:20:07.547901 9152 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:07.544714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.2.246.65:45291)
0927 15:20:07.545565 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.545577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.545674 (+ 97us) server_negotiation.cc:407] Connection header received
0927 15:20:07.545944 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.545952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.546181 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.546356 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.547438 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53616: BlockingRecv error: recv got EOF from 127.0.0.1:53616 (error 108)
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":367,"threads_started":1}
W20260927 15:20:07.548867 3033 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc)
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.551331 3033 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.553710 3033 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:20:07.573088 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.555822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.2.246.65:45291)
0927 15:20:07.556138 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.556146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.556443 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:20:07.556608 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.556613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.556694 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.556830 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.557975 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.559099 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.560395 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.561185 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.561335 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.565391 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.565411 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.565428 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.565490 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.569422 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.570442 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.570452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.570454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.570633 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.571181 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.571185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.571187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.571677 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.571997 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.572332 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.572716 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 15:20:07.582067 9152 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:07.579613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.2.246.65:45291)
0927 15:20:07.579913 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.579921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.580152 (+ 231us) server_negotiation.cc:407] Connection header received
0927 15:20:07.580240 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.580245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.580350 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.580577 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.581675 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53630: BlockingRecv error: recv got EOF from 127.0.0.1:53630 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260927 15:20:07.582119 3033 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.584014 3033 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.585358 3033 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:20:07.603379 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.587313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.2.246.65:45291)
0927 15:20:07.587594 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.587601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.587883 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:20:07.588148 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.588158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.588229 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.588345 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.589349 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.590248 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.591517 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.591945 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.592084 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.597359 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.597366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.597370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.597434 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.600548 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.601346 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.601352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.601356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.601557 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.601969 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.601976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.601980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.602275 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.602560 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.602844 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.603055 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 15:20:07.608839 9152 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:07.606784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.2.246.65:45291)
0927 15:20:07.607053 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.607059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.607274 (+ 215us) server_negotiation.cc:407] Connection header received
0927 15:20:07.607493 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.607497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.607578 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.607685 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.608556 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53654: BlockingRecv error: recv got EOF from 127.0.0.1:53654 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:20:07.609030 3033 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522351366233-3033-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.610854 3033 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:07.612615 3033 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:20:07.631418 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.614400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.2.246.65:45291)
0927 15:20:07.614762 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.614771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.615014 (+ 243us) server_negotiation.cc:407] Connection header received
0927 15:20:07.615169 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.615175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.615260 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.615387 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.616741 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.617579 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.619217 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.619753 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.619931 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.624378 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.624386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.624391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.624448 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.627745 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.628634 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.628639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.628641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.628876 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.629374 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.629379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.629381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.629738 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.630372 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.630655 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.630964 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 15:20:07.633530 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8748
I20260927 15:20:07.648430 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.634428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.2.246.65:45291)
0927 15:20:07.634608 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.634612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.634944 (+ 332us) server_negotiation.cc:407] Connection header received
0927 15:20:07.635064 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.635068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.635144 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.635269 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.636254 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.637033 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.638458 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.638982 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.639108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.641408 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.641414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.641418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.641463 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.644892 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.645877 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.645884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.645887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.646104 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.646679 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.646685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.646688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.647085 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.647346 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.647738 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.648071 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:20:07.649266 8871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:07.748837 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8748
I20260927 15:20:07.768950 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8882
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:20:07.787843 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.770278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.2.246.66:43765)
0927 15:20:07.770996 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.771004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.771033 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:20:07.771100 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.771107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.771341 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.771585 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.772690 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.773820 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.775268 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.775903 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.776068 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.780726 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.780739 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.780756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.780791 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.784005 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.785273 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.785281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.785283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.785453 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.786198 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.786202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.786204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.786582 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.786833 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.787307 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.787540 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":361,"threads_started":1}
I20260927 15:20:07.790172 9005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:07.884388 9010 heartbeater.cc:503] Master 127.2.246.126:37825 was elected leader, sending a full tablet report...
I20260927 15:20:07.890067 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8882
I20260927 15:20:07.906978 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9016
Sep 27 15:20:07 dist-test-slave-14jq krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:20:07.926340 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:07.908411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.2.246.67:34489)
0927 15:20:07.909152 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:07.909160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:07.909195 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:20:07.909271 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:07.909279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:07.909522 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:07.909682 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:07.911013 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.912018 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.913525 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:07.914183 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:07.914295 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:07.919200 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:07.919221 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:07.919249 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:07.919309 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:07.922810 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.923798 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.923806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.923809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.923978 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:07.924474 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:07.924479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:07.924481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:07.924942 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:07.925222 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:07.925582 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:20:07.925930 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":379,"threads_started":1}
I20260927 15:20:07.928102 9139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:08.024307 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9016
I20260927 15:20:08.041182 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 8655
I20260927 15:20:08.042642 8717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:08.147589 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 8655
2026-09-27T15:20:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4653 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:08 dist-test-slave-14jq krb5kdc[9172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:08 dist-test-slave-14jq krb5kdc[9172](info): set up 2 sockets
Sep 27 15:20:08 dist-test-slave-14jq krb5kdc[9172](info): commencing operation
krb5kdc: starting...
W20260927 15:20:09.294790 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:20:09 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab.
Entry for principal oryx/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab.
I20260927 15:20:09.558835 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35349
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab
--principal=oryx/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:09.975378 9185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:09.975777 9185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:09.975953 9185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:09.984292 9185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:09.984452 9185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:09.984567 9185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:09.984690 9185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:09.984760 9185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:09.996320 9185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab
--principal=oryx/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:09.998221 9185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:09.999915 9185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:10.015622 9191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:10.015698 9193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:10.015684 9190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:10.017583 9185 server_base.cc:1034] running on GCE node
I20260927 15:20:10.018656 9185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:10.020459 9185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:10.021760 9185 hybrid_clock.cc:648] HybridClock initialized: now 1790522410021705 us; error 70 us; skew 500 ppm
I20260927 15:20:10.022257 9185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:10 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522410, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:10.026194 9185 init.cc:377] Logged in from keytab as oryx/127.2.246.126@KRBTEST.COM (short username oryx)
I20260927 15:20:10.028133 9185 webserver.cc:460] Webserver started at http://127.2.246.126:39829/ using document root <none> and password file <none>
I20260927 15:20:10.028976 9185 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:10.029136 9185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:10.029469 9185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:10.032862 9185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "01abce2181f74267bccc54e24211dd8d"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "973592f093fb4c90ed4549c073bceccc"
server_key_iv: "837a6dff182d78e15b63d6fe7f92cc6a"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.033715 9185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "01abce2181f74267bccc54e24211dd8d"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "973592f093fb4c90ed4549c073bceccc"
server_key_iv: "837a6dff182d78e15b63d6fe7f92cc6a"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.040123 9185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 15:20:10.045051 9200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:10.047327 9185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:10.047708 9185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "01abce2181f74267bccc54e24211dd8d"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "973592f093fb4c90ed4549c073bceccc"
server_key_iv: "837a6dff182d78e15b63d6fe7f92cc6a"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.047960 9185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:10.080978 9185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:10.087172 9185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:10.087535 9185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:10.110730 9185 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:35349
I20260927 15:20:10.110800 9251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:35349 every 8 connection(s)
I20260927 15:20:10.112413 9185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:10.116633 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9185
I20260927 15:20:10.116883 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:10.117563 3033 external_mini_cluster.cc:1428] Setting key bd1fb8dab9d166bac76f63ea5996c6e6
I20260927 15:20:10.117812 9252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:20:10 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:10.136531 9252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Bootstrap starting.
I20260927 15:20:10.140826 9252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:10.142448 9252 log.cc:824] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:10.147001 9252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: No bootstrap required, opened a new log
I20260927 15:20:10.147619 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:10.120034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.2.246.126:35349)
0927 15:20:10.121299 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:10.121327 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:10.122012 (+ 685us) server_negotiation.cc:407] Connection header received
0927 15:20:10.124299 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:10.124329 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:10.124871 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:10.125290 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:10.127659 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.129135 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.131194 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.132133 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.132326 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:10.137885 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:10.137902 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:10.137939 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:10.137999 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:10.141571 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:10.142629 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:10.142639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:10.142649 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:10.142903 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:10.143515 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:10.143520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:10.143523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:10.144503 (+ 980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:10.144906 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:10.145338 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 15:20:10.145942 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":516,"threads_started":1}
I20260927 15:20:10.157701 9252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } }
I20260927 15:20:10.158303 9252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:10.158389 9252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01abce2181f74267bccc54e24211dd8d, State: Initialized, Role: FOLLOWER
I20260927 15:20:10.159053 9252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } }
I20260927 15:20:10.159291 9252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:10.159404 9252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:10.159547 9252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:10.162135 9252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } }
I20260927 15:20:10.162787 9252 leader_election.cc:302] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01abce2181f74267bccc54e24211dd8d; no voters:
I20260927 15:20:10.163841 9252 leader_election.cc:288] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:10.164400 9257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:10.167320 9252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:10.167876 9257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 LEADER]: Becoming Leader. State: Replica: 01abce2181f74267bccc54e24211dd8d, State: Running, Role: LEADER
I20260927 15:20:10.168629 9257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } }
I20260927 15:20:10.172557 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01abce2181f74267bccc54e24211dd8d. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } }
I20260927 15:20:10.173204 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:10.173771 9258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } }
I20260927 15:20:10.174474 9258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:10.175681 9267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:10.183226 9267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:10.197489 9267 catalog_manager.cc:1347] Generated new cluster ID: bfa39b6ae0a64801b3a56667a890e0de
I20260927 15:20:10.197706 9267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:10.215852 9267 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:10.217621 9267 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:10.230669 9267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Generated new TSK 0
I20260927 15:20:10.231957 9267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
I20260927 15:20:10.303627 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:10.749460 9278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:10.750116 9278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:10.750360 9278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:10.759949 9278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:10.760128 9278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:10.760319 9278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:10.772913 9278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:10.774999 9278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:10.776829 9278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:10.794400 9283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:10.794498 9286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:10.794577 9284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:10.795773 9278 server_base.cc:1034] running on GCE node
I20260927 15:20:10.796698 9278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:10.798075 9278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:10.799315 9278 hybrid_clock.cc:648] HybridClock initialized: now 1790522410799229 us; error 74 us; skew 500 ppm
I20260927 15:20:10.799916 9278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:10 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522410, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:10.808403 9278 init.cc:377] Logged in from keytab as oryx/127.2.246.65@KRBTEST.COM (short username oryx)
I20260927 15:20:10.810701 9278 webserver.cc:460] Webserver started at http://127.2.246.65:46025/ using document root <none> and password file <none>
I20260927 15:20:10.811655 9278 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:10.811826 9278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:10.812243 9278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:10.816063 9278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "c11e7745dc864f68b0cad8fefdf23540"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "67cbb7701583ae7b8d51d4cb44310336"
server_key_iv: "71b9f73e0d3182ef7c438634af7a60d6"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.817062 9278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "c11e7745dc864f68b0cad8fefdf23540"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "67cbb7701583ae7b8d51d4cb44310336"
server_key_iv: "71b9f73e0d3182ef7c438634af7a60d6"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.824368 9278 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:20:10.829319 9293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:10.831525 9278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:20:10.831846 9278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c11e7745dc864f68b0cad8fefdf23540"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "67cbb7701583ae7b8d51d4cb44310336"
server_key_iv: "71b9f73e0d3182ef7c438634af7a60d6"
server_key_version: "encryptionkey@0"
I20260927 15:20:10.832114 9278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:10.880303 9278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:10.887049 9278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:10.887465 9278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:10.888962 9278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:10.890951 9278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:10.891045 9278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:10.891155 9278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:10.891212 9278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:10.936122 9278 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:46669
I20260927 15:20:10.936168 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:46669 every 8 connection(s)
I20260927 15:20:10.937913 9278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:10.944344 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9278
I20260927 15:20:10.944590 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:10.945277 3033 external_mini_cluster.cc:1428] Setting key 4de19d5a3fa98451a77bfee16e1b291c
Sep 27 15:20:10 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522410, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:10.965795 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:10.941228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:46881 (local address 127.2.246.126:35349)
0927 15:20:10.941786 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:10.941794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:10.943136 (+ 1342us) server_negotiation.cc:407] Connection header received
0927 15:20:10.945541 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:10.945548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:10.945650 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:10.945816 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:10.948382 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.949631 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.951907 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.952720 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.952919 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:10.958591 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:10.958597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:10.958601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:10.958644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:10.961341 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:10.962790 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:10.962795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:10.962797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:10.962949 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:10.963620 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:10.963623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:10.963626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:10.963916 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:10.964037 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:10.965276 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0927 15:20:10.965531 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260927 15:20:10.967967 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:10.941900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:46881)
0927 15:20:10.942912 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:10.942967 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:10.945115 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:10.946448 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:10.946461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:10.947169 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:10.948146 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:10.948171 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.949969 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.949975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:10.951664 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:10.951691 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:10.953102 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:10.953110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:10.953562 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:10.954736 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:10.954767 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:10.958125 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:10.961601 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:10.961619 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:10.961649 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:10.962585 (+ 936us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:10.963105 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:10.963113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:10.963120 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:10.963461 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:10.964072 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:10.964086 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:10.964727 (+ 641us) client_negotiation.cc:769] Sending connection context
0927 15:20:10.965132 (+ 405us) client_negotiation.cc:240] Negotiation successful
0927 15:20:10.965918 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":364,"threads_started":1}
I20260927 15:20:10.970175 9406 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:10.970559 9406 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:20:10.971462 9406 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:10.973893 9217 ts_manager.cc:194] Registered new tserver with Master: c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.66@KRBTEST.COM" created.
I20260927 15:20:10.976137 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:46881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
I20260927 15:20:11.006042 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:11.445287 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:11.445675 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:11.445865 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:11.453644 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:11.453801 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:11.454016 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:11.464126 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:11.465663 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:11.467131 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:11.482617 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:11.482987 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:11.482987 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:11.484294 9412 server_base.cc:1034] running on GCE node
I20260927 15:20:11.484983 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:11.486162 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:11.487511 9412 hybrid_clock.cc:648] HybridClock initialized: now 1790522411487420 us; error 83 us; skew 500 ppm
I20260927 15:20:11.487891 9412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:11 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:11.497439 9412 init.cc:377] Logged in from keytab as oryx/127.2.246.66@KRBTEST.COM (short username oryx)
I20260927 15:20:11.499611 9412 webserver.cc:460] Webserver started at http://127.2.246.66:46513/ using document root <none> and password file <none>
I20260927 15:20:11.500649 9412 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:11.500811 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:11.501304 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:11.504977 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "b33c24343375466f8861a09c8e6dc5d1"
format_stamp: "Formatted at 2026-09-27 15:20:11 on dist-test-slave-14jq"
server_key: "00b1e0932fa09a69f4ad657b785e2de7"
server_key_iv: "b0ec4fafdb14afbeb15129f19fe605ed"
server_key_version: "encryptionkey@0"
I20260927 15:20:11.505939 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "b33c24343375466f8861a09c8e6dc5d1"
format_stamp: "Formatted at 2026-09-27 15:20:11 on dist-test-slave-14jq"
server_key: "00b1e0932fa09a69f4ad657b785e2de7"
server_key_iv: "b0ec4fafdb14afbeb15129f19fe605ed"
server_key_version: "encryptionkey@0"
I20260927 15:20:11.512274 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:11.517237 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:11.519564 9412 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:20:11.519827 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "b33c24343375466f8861a09c8e6dc5d1"
format_stamp: "Formatted at 2026-09-27 15:20:11 on dist-test-slave-14jq"
server_key: "00b1e0932fa09a69f4ad657b785e2de7"
server_key_iv: "b0ec4fafdb14afbeb15129f19fe605ed"
server_key_version: "encryptionkey@0"
I20260927 15:20:11.520095 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:11.544538 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:11.549603 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:11.550119 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:11.551342 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:11.553081 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:11.553185 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:11.553318 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:11.553406 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:11.593215 9412 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:43861
I20260927 15:20:11.593269 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:43861 every 8 connection(s)
I20260927 15:20:11.594717 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:11.604914 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9412
I20260927 15:20:11.605149 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:11.605782 3033 external_mini_cluster.cc:1428] Setting key 2a9bcab9058ab043de874f51527407cd
Sep 27 15:20:11 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:11.623781 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:11.598016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:43947 (local address 127.2.246.126:35349)
0927 15:20:11.598604 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:11.598612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:11.600017 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 15:20:11.602384 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:11.602390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:11.602500 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:11.602649 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:11.604643 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:11.605905 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:11.607911 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:11.608503 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:11.608625 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:11.614075 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:11.614085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:11.614094 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:11.614148 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:11.617723 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:11.619296 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:11.619301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:11.619304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:11.619482 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:11.620245 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:11.620249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:11.620252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:11.620591 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:11.620795 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:11.622936 (+ 2141us) server_negotiation.cc:299] Negotiation successful
0927 15:20:11.623230 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":314,"threads_started":1}
I20260927 15:20:11.626718 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:11.598805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:43947)
0927 15:20:11.599807 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:11.599867 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:11.602002 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:11.603030 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:11.603039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:11.603703 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:11.604367 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:11.604387 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:11.606119 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:11.606123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:11.607684 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:11.607698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:11.608768 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:11.608775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:11.609154 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:11.610127 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:11.610171 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:11.613491 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:11.618046 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:11.618065 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:11.618097 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:11.619083 (+ 986us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:11.619647 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:11.619656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:11.619669 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:11.620039 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:11.620771 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:11.620785 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:11.622301 (+ 1516us) client_negotiation.cc:769] Sending connection context
0927 15:20:11.622749 (+ 448us) client_negotiation.cc:240] Negotiation successful
0927 15:20:11.623915 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":372,"threads_started":1}
I20260927 15:20:11.629690 9540 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:11.630204 9540 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:11.631636 9540 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:11.634701 9217 ts_manager.cc:194] Registered new tserver with Master: b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:11.636116 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:43947
WARNING: no policy specified for oryx/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
I20260927 15:20:11.682361 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:11.980680 9406 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
W20260927 15:20:12.130434 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:12.130832 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:12.131021 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:12.140627 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:12.140810 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:12.141057 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:12.154809 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:12.156706 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:12.158533 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:12.177382 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:12.177429 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:12.177399 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:12.178918 9546 server_base.cc:1034] running on GCE node
I20260927 15:20:12.179875 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:12.181314 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:12.182600 9546 hybrid_clock.cc:648] HybridClock initialized: now 1790522412182528 us; error 64 us; skew 500 ppm
I20260927 15:20:12.183103 9546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522412, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:12.187422 9546 init.cc:377] Logged in from keytab as oryx/127.2.246.67@KRBTEST.COM (short username oryx)
I20260927 15:20:12.190002 9546 webserver.cc:460] Webserver started at http://127.2.246.67:35257/ using document root <none> and password file <none>
I20260927 15:20:12.191008 9546 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:12.191177 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:12.191929 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:12.196162 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "c607a92dcead4686badf99847336fcd8"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "dc600d168b7a30aed80fb98fa02ac9a1"
server_key_iv: "124cca41675fd1ef27fd41f4a904ed1f"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.197279 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "c607a92dcead4686badf99847336fcd8"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "dc600d168b7a30aed80fb98fa02ac9a1"
server_key_iv: "124cca41675fd1ef27fd41f4a904ed1f"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.205546 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260927 15:20:12.211215 9561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:12.213384 9546 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:20:12.213660 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "c607a92dcead4686badf99847336fcd8"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "dc600d168b7a30aed80fb98fa02ac9a1"
server_key_iv: "124cca41675fd1ef27fd41f4a904ed1f"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.213963 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:12.235203 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:12.241036 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:12.241413 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:12.242782 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:12.244560 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:12.244650 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:12.244818 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:12.244875 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:12.300413 9546 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:35785
I20260927 15:20:12.300490 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:35785 every 8 connection(s)
I20260927 15:20:12.302287 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:12.312322 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9546
I20260927 15:20:12.312597 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:12.313300 3033 external_mini_cluster.cc:1428] Setting key f64a273ca1501a84f22593a58a00e38b
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522412, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:12.327921 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.305732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:60085 (local address 127.2.246.126:35349)
0927 15:20:12.306316 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:12.306322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:12.307738 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 15:20:12.309726 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:12.309731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:12.309883 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:12.310016 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:12.311777 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.312627 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.315669 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.316271 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.316390 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.321296 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:12.321301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:12.321305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:12.321345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:12.324251 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.325448 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.325452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.325455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.325578 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.326123 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.326126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.326128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.326392 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:12.326512 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:12.327436 (+ 924us) server_negotiation.cc:299] Negotiation successful
0927 15:20:12.327603 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
I20260927 15:20:12.329737 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.306552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:60085)
0927 15:20:12.307528 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:12.307587 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:12.309402 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:12.310294 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:12.310303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:12.310884 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:12.311562 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.311579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.312959 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.312969 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.315414 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.315440 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.316582 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.316589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.316916 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.318011 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:12.318043 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:12.320903 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:12.324487 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.324498 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.324518 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.325238 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.325698 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.325702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.325705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.326001 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.326521 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:12.326529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:12.327018 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 15:20:12.327321 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 15:20:12.327836 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":344,"threads_started":1}
I20260927 15:20:12.331651 9674 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:12.332063 9674 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:12.332924 9674 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:12.334744 9217 ts_manager.cc:194] Registered new tserver with Master: c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785)
I20260927 15:20:12.335815 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:60085
I20260927 15:20:12.344331 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.125.keytab.
Entry for principal oryx/127.2.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.125.keytab.
I20260927 15:20:12.402997 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:37729
--webserver_interface=127.2.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.125.keytab
--principal=oryx/127.2.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35349,127.2.246.125:37729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:12.640057 9540 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
W20260927 15:20:12.771704 9680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:12.772030 9680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:12.772162 9680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:12.779808 9680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:12.779944 9680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:12.779995 9680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:12.780045 9680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:12.780081 9680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:12.790937 9680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35349,127.2.246.125:37729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.125.keytab
--principal=oryx/127.2.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
W20260927 15:20:12.791356 9680 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.246.126:35349,127.2.246.125:37729'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:20:12.792589 9680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:12.794096 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:12.809173 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:12.809330 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:12.809332 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:12.810570 9680 server_base.cc:1034] running on GCE node
I20260927 15:20:12.811718 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:12.813640 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:12.814978 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790522412814875 us; error 75 us; skew 500 ppm
I20260927 15:20:12.815466 9680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522412, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:12.819599 9680 init.cc:377] Logged in from keytab as oryx/127.2.246.125@KRBTEST.COM (short username oryx)
I20260927 15:20:12.821664 9680 webserver.cc:460] Webserver started at http://127.2.246.125:46495/ using document root <none> and password file <none>
I20260927 15:20:12.822698 9680 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:12.822846 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:12.823225 9680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:12.827276 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/instance:
uuid: "1503936f32ae4a87ac1c183c9a1b0308"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "45ba62bdbe13a6c70f34024603cbd08c"
server_key_iv: "70e18218c2e881e709d46fb036105d45"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.828877 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal/instance:
uuid: "1503936f32ae4a87ac1c183c9a1b0308"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "45ba62bdbe13a6c70f34024603cbd08c"
server_key_iv: "70e18218c2e881e709d46fb036105d45"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.836766 9680 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:20:12.842275 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:12.844491 9680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:20:12.844870 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "1503936f32ae4a87ac1c183c9a1b0308"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "45ba62bdbe13a6c70f34024603cbd08c"
server_key_iv: "70e18218c2e881e709d46fb036105d45"
server_key_version: "encryptionkey@0"
I20260927 15:20:12.845168 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:12.889866 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:12.894927 9680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:12.895251 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:12.918866 9680 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.125:37729
I20260927 15:20:12.918912 9746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.125:37729 every 8 connection(s)
I20260927 15:20:12.920821 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
I20260927 15:20:12.922153 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9680
I20260927 15:20:12.927400 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:12.936898 9747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
I20260927 15:20:12.950961 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.925032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.2.246.125:37729)
0927 15:20:12.926248 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:12.926265 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:12.926317 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:20:12.928312 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:12.928344 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:12.928879 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:12.929400 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:12.931184 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.932787 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.934301 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.935088 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.935348 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.940314 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:12.940324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:12.940343 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:12.940391 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:12.944924 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.946391 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.946470 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.946478 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.946669 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.947164 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.947170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.947181 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.947984 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:12.948334 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:12.948855 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:20:12.949344 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":557,"threads_started":1}
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522412, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.125@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:12.961894 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.940780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57208)
0927 15:20:12.942262 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:12.942334 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:12.942539 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:12.943643 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:12.943653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:12.944700 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:12.945166 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.945186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.946967 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.946971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.948877 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.948891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.949859 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.949867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.950098 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.950914 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:12.950957 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:12.954148 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:12.958551 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.958565 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.958572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.959148 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.959736 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.959740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.959743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.959894 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.960722 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:12.960729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:12.960915 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 15:20:12.961157 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 15:20:12.961494 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":831,"threads_started":1}
I20260927 15:20:12.961973 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.940837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.2.246.126:35349)
0927 15:20:12.943230 (+ 2393us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:12.943236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:12.943258 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:12.943340 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:12.943344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:12.943419 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:12.943542 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:12.945383 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.946718 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.949084 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.949566 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.949723 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.954438 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:12.954446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:12.954452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:12.954512 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:12.958316 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.959359 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.959365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.959368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.959579 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.960063 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.960070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.960074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.960547 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:12.960741 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:12.961313 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 15:20:12.961559 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2285,"thread_start_us":2112,"threads_started":1}
I20260927 15:20:12.964453 9747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:20:12 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522412, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.125@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
I20260927 15:20:12.979277 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.965454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39718)
0927 15:20:12.965746 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:12.965763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:12.965936 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:12.966285 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:12.966288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:12.966614 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:12.966771 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.966779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.968069 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.968073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.969210 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:12.969222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.970193 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.970198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:12.970382 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.971206 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:12.971223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:12.973606 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:12.976799 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.976803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.976806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.977225 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.977724 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:12.977727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:12.977730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:12.977914 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:12.978499 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:12.978502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:12.978650 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 15:20:12.978773 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:20:12.979029 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 15:20:12.979655 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:12.965650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.2.246.125:37729)
0927 15:20:12.965913 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:12.965925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:12.965946 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:20:12.966059 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:12.966063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:12.966130 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:12.966258 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:12.966939 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.967886 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.969402 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:12.970005 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:12.970146 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:12.973909 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:12.973922 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:12.973927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:12.973969 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:12.976629 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.977384 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.977388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.977391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.977598 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:12.978054 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:12.978058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:12.978061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:12.978376 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:12.978509 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:12.978887 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 15:20:12.979192 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":68}
I20260927 15:20:12.985450 9747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Bootstrap starting.
I20260927 15:20:12.989352 9747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:12.990993 9747 log.cc:824] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:12.994881 9747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: No bootstrap required, opened a new log
I20260927 15:20:13.003264 9747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } }
I20260927 15:20:13.003707 9747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:13.003791 9747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1503936f32ae4a87ac1c183c9a1b0308, State: Initialized, Role: FOLLOWER
I20260927 15:20:13.004462 9747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } }
I20260927 15:20:13.005550 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } } }
I20260927 15:20:13.006074 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:13.007335 9747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:20:13.018055 9765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:13.018244 9765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:13.022207 9680 master_runner.cc:428] Detected that this master 1503936f32ae4a87ac1c183c9a1b0308 is joining an existing cluster
I20260927 15:20:13.022361 9680 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.246.125:37729
I20260927 15:20:13.045733 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.028348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.2.246.126:35349)
0927 15:20:13.028669 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.028680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.029296 (+ 616us) server_negotiation.cc:407] Connection header received
0927 15:20:13.029645 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.029652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.029775 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.029902 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.032270 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.033463 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.035904 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.036653 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.036772 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.039686 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.039692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.039696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.039732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.043024 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.043706 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.043711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.043716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.043848 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.044219 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.044227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.044230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.044542 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.044697 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.045118 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.045362 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:20:13.046120 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.029506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.2.246.125:37729)
0927 15:20:13.029763 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.029771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.030518 (+ 747us) server_negotiation.cc:407] Connection header received
0927 15:20:13.030707 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.030713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.030782 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.030909 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.032217 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.032952 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.034502 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.035022 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.035170 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.038019 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.038026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.038032 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.038076 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.042434 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.043418 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.043427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.043433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.043648 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.044128 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.044133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.044136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.044549 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.044707 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.045375 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.045613 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:20:13.046667 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.029285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39722)
0927 15:20:13.030341 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.030418 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.030585 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.030918 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.030922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.031679 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.032042 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.032052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.033115 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.033120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.034349 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.034371 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.035222 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.035228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.035437 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.036098 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.036115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.037722 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.042637 (+ 4915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.042643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.042650 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.043226 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.043794 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.043799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.043804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.043968 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.044759 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.044764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.044927 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.045070 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.046158 (+ 1088us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":734,"thread_start_us":697,"threads_started":1}
I20260927 15:20:13.047101 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.028280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57222)
0927 15:20:13.029132 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.029157 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.029454 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.030293 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.030300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.031452 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.032009 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.032032 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.033719 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.033725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.035687 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.035704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.037158 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.037166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.037379 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.038267 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.038285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.039437 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.043198 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.043203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.043206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.043588 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.043955 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.043961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.043965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.044111 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.044656 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.044661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.044841 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.045018 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.045367 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":39,"thread_start_us":444,"threads_started":1}
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522410, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.126@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
I20260927 15:20:13.075060 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.055537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.2.246.125:37729)
0927 15:20:13.055760 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.055769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.056603 (+ 834us) server_negotiation.cc:407] Connection header received
0927 15:20:13.056990 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.056994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.057102 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.057210 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.059024 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.060202 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.062577 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.063462 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.063643 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.068781 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.068787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.068791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.068831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.072048 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.072904 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.072909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.072913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.073053 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.073459 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.073465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.073473 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.073765 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.073978 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.074440 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.074670 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:20:13.075110 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.055309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39724)
0927 15:20:13.056362 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.056426 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.056698 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.057357 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.057367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.058474 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.058806 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.058822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.060393 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.060400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.062321 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.062341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.063715 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.063724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.063968 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.064968 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.064997 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.068436 (+ 3439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.072239 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.072273 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.072285 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.072758 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.073176 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.073181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.073185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.073344 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.073940 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.073950 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.074138 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.074329 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.074688 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":507,"threads_started":1}
I20260927 15:20:13.077550 9217 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.246.125:37729: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308"
member_type: NON_VOTER
last_known_addr {
host: "127.2.246.125"
port: 37729
}
attrs {
promote: true
}
}
dest_uuid: "01abce2181f74267bccc54e24211dd8d"
cas_config_opid_index: -1
I20260927 15:20:13.078617 9217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.2.246.124@KRBTEST.COM; defaulting to no policy
I20260927 15:20:13.087877 9722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:13.087097 9777 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.124@KRBTEST.COM" created.
I20260927 15:20:13.092660 9777 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.246.125:37729
I20260927 15:20:13.093219 9776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } }
I20260927 15:20:13.093604 9680 master.cc:537] Master@127.2.246.125:37729 shutting down...
I20260927 15:20:13.093705 9776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:13.102628 9722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:20:13.104367 9777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:20:13.108413 9204 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 15:20:13.115855 9204 consensus_peers.cc:487] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d -> Peer 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125:37729): Couldn't send request to peer 1503936f32ae4a87ac1c183c9a1b0308. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:20:13.120829 9680 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:13.121426 9680 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:13.121632 9680 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.124.keytab.
Entry for principal oryx/127.2.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.124.keytab.
I20260927 15:20:13.138813 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:41739
--webserver_interface=127.2.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.124.keytab
--principal=oryx/127.2.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:13.141403 9680 master.cc:552] Master@127.2.246.125:37729 shutdown complete.
I20260927 15:20:13.141516 9680 master_runner.cc:305] Clearing existing system tablet
I20260927 15:20:13.142023 9680 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:13.142128 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:13.146849 9680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:20:13.148612 9783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:13.148988 9680 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:13.149253 9680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 15:20:13.149338 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "1503936f32ae4a87ac1c183c9a1b0308"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "45ba62bdbe13a6c70f34024603cbd08c"
server_key_iv: "70e18218c2e881e709d46fb036105d45"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.153390 9680 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:13.158525 9680 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:20:13.158752 9680 log.cc:1196] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:20:13.159219 9680 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Deleting consensus metadata
I20260927 15:20:13.159494 9680 master_runner.cc:315] Copying system tablet from 127.2.246.126:35349
I20260927 15:20:13.162216 9680 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: tablet copy: Beginning tablet copy session from remote peer at address 127.2.246.126:35349
I20260927 15:20:13.179606 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.163077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.2.246.126:35349)
0927 15:20:13.163299 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.163309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.166614 (+ 3305us) server_negotiation.cc:407] Connection header received
0927 15:20:13.166863 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.166868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.166936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.167050 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.168150 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.169124 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.170531 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.170985 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.171142 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.173778 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.173784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.173788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.173823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.176708 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.177449 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.177455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.177459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.177590 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.178071 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.178077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.178080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.178367 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.178543 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.178933 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.179179 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:20:13.179644 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.163051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57238)
0927 15:20:13.166454 (+ 3403us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.166479 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.166692 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.167069 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.167073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.167707 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.167969 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.167979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.169296 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.169304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.170376 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.170388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.171405 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.171410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.171534 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.172529 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.172548 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.173506 (+ 958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.176895 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.176901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.176905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.177298 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.177709 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.177715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.177718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.177950 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.178509 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.178515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.178691 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.178831 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.179192 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3255,"thread_start_us":3090,"threads_started":1}
I20260927 15:20:13.180817 9237 tablet_copy_service.cc:140] P 01abce2181f74267bccc54e24211dd8d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1503936f32ae4a87ac1c183c9a1b0308 ({username='oryx', principal='oryx/127.2.246.125@KRBTEST.COM'} at 127.0.0.1:57238)
I20260927 15:20:13.181099 9237 tablet_copy_service.cc:161] P 01abce2181f74267bccc54e24211dd8d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1503936f32ae4a87ac1c183c9a1b0308 at {username='oryx', principal='oryx/127.2.246.125@KRBTEST.COM'} at 127.0.0.1:57238: session id = 1503936f32ae4a87ac1c183c9a1b0308-00000000000000000000000000000000
I20260927 15:20:13.184619 9237 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:20:13.186985 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:20:13.192833 9680 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: tablet copy: Starting download of 0 data blocks...
I20260927 15:20:13.193190 9680 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: tablet copy: Starting download of 1 WAL segments...
I20260927 15:20:13.195881 9680 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:20:13.199571 9237 tablet_copy_service.cc:342] P 01abce2181f74267bccc54e24211dd8d: Request end of tablet copy session 1503936f32ae4a87ac1c183c9a1b0308-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.246.125@KRBTEST.COM'} at 127.0.0.1:57238
I20260927 15:20:13.199793 9237 tablet_copy_service.cc:433] P 01abce2181f74267bccc54e24211dd8d: ending tablet copy session 1503936f32ae4a87ac1c183c9a1b0308-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1503936f32ae4a87ac1c183c9a1b0308
I20260927 15:20:13.211632 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:13.214809 9790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.226565 9791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.227828 9793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:13.234310 9680 server_base.cc:1034] running on GCE node
I20260927 15:20:13.234859 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:13.235754 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:13.237007 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790522413236933 us; error 46 us; skew 500 ppm
I20260927 15:20:13.237270 9680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:13.239429 9680 init.cc:377] Logged in from keytab as oryx/127.2.246.125@KRBTEST.COM (short username oryx)
I20260927 15:20:13.242769 9680 webserver.cc:460] Webserver started at http://127.2.246.125:45847/ using document root <none> and password file <none>
I20260927 15:20:13.243184 9680 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:13.243280 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:13.245636 9680 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 15:20:13.247412 9800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:13.247844 9680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 15:20:13.247996 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "1503936f32ae4a87ac1c183c9a1b0308"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "45ba62bdbe13a6c70f34024603cbd08c"
server_key_iv: "70e18218c2e881e709d46fb036105d45"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.248152 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:13.288834 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:13.292425 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:13.337926 9680 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.125:37729
I20260927 15:20:13.337949 9851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.125:37729 every 8 connection(s)
I20260927 15:20:13.338850 9674 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
I20260927 15:20:13.339035 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
I20260927 15:20:13.348975 9852 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:20:13.352253 9852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Bootstrap starting.
I20260927 15:20:13.366659 9852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:20:13.367174 9852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: Bootstrap complete.
I20260927 15:20:13.368202 9852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } }
I20260927 15:20:13.368479 9852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1503936f32ae4a87ac1c183c9a1b0308, State: Initialized, Role: LEARNER
I20260927 15:20:13.368718 9852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } }
I20260927 15:20:13.369778 9855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } }
I20260927 15:20:13.370162 9852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:13.370270 9855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:20:13.382377 9866 catalog_manager.cc:1259] Loaded cluster ID: bfa39b6ae0a64801b3a56667a890e0de
I20260927 15:20:13.382506 9866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: loading cluster ID for follower catalog manager: success
I20260927 15:20:13.385165 9866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: acquiring CA information for follower catalog manager: success
I20260927 15:20:13.387436 9866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:20:13.640583 9782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:13.640897 9782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:13.641028 9782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:13.649277 9782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:13.649441 9782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:13.649507 9782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:13.649571 9782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:13.649617 9782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:13.659600 9782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.124.keytab
--principal=oryx/127.2.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:13.660956 9782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:13.662248 9782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:13.676952 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.677183 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.677328 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:13.684033 9782 server_base.cc:1034] running on GCE node
I20260927 15:20:13.684970 9782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:13.686740 9782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:13.687991 9782 hybrid_clock.cc:648] HybridClock initialized: now 1790522413687933 us; error 77 us; skew 500 ppm
I20260927 15:20:13.688432 9782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:13.692121 9782 init.cc:377] Logged in from keytab as oryx/127.2.246.124@KRBTEST.COM (short username oryx)
I20260927 15:20:13.694579 9782 webserver.cc:460] Webserver started at http://127.2.246.124:41301/ using document root <none> and password file <none>
I20260927 15:20:13.695993 9782 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:13.696141 9782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:13.696478 9782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:13.699671 9782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/instance:
uuid: "276a84cd677549cbacf13a50293fdfb6"
format_stamp: "Formatted at 2026-09-27 15:20:13 on dist-test-slave-14jq"
server_key: "1c58c35a76ba319425b591cf110a7980"
server_key_iv: "ab86289d7f525e3274226e0405f75a8b"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.700520 9782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal/instance:
uuid: "276a84cd677549cbacf13a50293fdfb6"
format_stamp: "Formatted at 2026-09-27 15:20:13 on dist-test-slave-14jq"
server_key: "1c58c35a76ba319425b591cf110a7980"
server_key_iv: "ab86289d7f525e3274226e0405f75a8b"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.701987 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.694745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.2.246.125:37729)
0927 15:20:13.695881 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.695888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.695917 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:20:13.695996 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.696000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.696077 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.696262 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:13.696814 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.697961 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.699669 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.700715 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.700886 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.701190 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.701392 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.701544 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":805,"threads_started":1}
I20260927 15:20:13.701968 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.694511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39738)
0927 15:20:13.695137 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.695159 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.695315 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.696304 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.696312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.696360 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:13.696618 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.696632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.698125 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.698131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.699479 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.699494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.700939 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.700946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.701158 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.701235 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.701364 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.701545 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":355,"threads_started":1}
I20260927 15:20:13.703277 9827 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 15:20:13.704056 9855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01abce2181f74267bccc54e24211dd8d. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } }
I20260927 15:20:13.704384 9855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:20:13.707671 9782 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 15:20:13.712576 9884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:13.714668 9782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:13.714915 9782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "276a84cd677549cbacf13a50293fdfb6"
format_stamp: "Formatted at 2026-09-27 15:20:13 on dist-test-slave-14jq"
server_key: "1c58c35a76ba319425b591cf110a7980"
server_key_iv: "ab86289d7f525e3274226e0405f75a8b"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.715137 9782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:13.738078 9782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:13.743641 9782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:13.743997 9782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:13.770714 9782 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.124:41739
I20260927 15:20:13.770785 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.124:41739 every 8 connection(s)
I20260927 15:20:13.772605 9782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
I20260927 15:20:13.775206 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9782
I20260927 15:20:13.778306 9936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:13.788232 9936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.124@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:13.804478 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.777869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.2.246.124:41739)
0927 15:20:13.778954 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.778971 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.779025 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:20:13.780751 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.780782 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.781252 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.781753 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.783431 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.785405 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.786759 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.787532 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.787759 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.794528 (+ 6769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.794539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.794553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.794594 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.798724 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.799712 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.799722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.799733 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.799975 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.800531 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.800535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.800537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.801327 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.801625 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.802210 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.802734 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"spinlock_wait_cycles":18304,"thread_start_us":536,"threads_started":1}
I20260927 15:20:13.812455 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.791907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.2.246.126:35349)
0927 15:20:13.793397 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.793401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.793416 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:20:13.793466 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.793470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.793526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.793627 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.795585 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.796630 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.799199 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.799793 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.800007 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.805307 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.805314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.805321 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.805371 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.809078 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.810243 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.810248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.810250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.810395 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.810910 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.810914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.810917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.811202 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.811337 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.811951 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.812127 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":540,"threads_started":1}
I20260927 15:20:13.812469 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.791779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57244)
0927 15:20:13.792939 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.793000 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.793172 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.793784 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.793795 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.794925 (+ 1130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.795343 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.795361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.796798 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.796811 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.798994 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.799014 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.800117 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.800126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.800384 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.801198 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.801222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.805025 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.809280 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.809293 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.809301 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.810052 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.810561 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.810566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.810569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.810758 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.811417 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.811425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.811623 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.811812 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.812130 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":406,"threads_started":1}
I20260927 15:20:13.814649 9936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.124@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
I20260927 15:20:13.832386 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.815850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.2.246.125:37729)
0927 15:20:13.816118 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.816128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.816154 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:20:13.816215 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.816221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.816315 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.816440 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.817292 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.818245 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.820020 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.820558 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.820724 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.825970 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.825978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.825983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.826035 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.829082 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.830161 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.830167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.830171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.830341 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.830772 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.830778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.830781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.831112 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.831289 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.831671 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.831961 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:20:13.832369 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.815541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39740)
0927 15:20:13.815811 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.815844 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.816000 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.816467 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.816472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.816953 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.817121 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.817130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.818402 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.818408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.819815 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.819848 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.820744 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.820751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.820973 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.821805 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.821825 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.825449 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.829340 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.829347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.829352 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.830006 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.830485 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.830490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.830494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.830652 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.831239 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.831244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.831413 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.831565 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.831962 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 15:20:13.834044 9936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.124@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
I20260927 15:20:13.852144 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.835014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.0.0.1:56214)
0927 15:20:13.835309 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.835332 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.835474 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.835943 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.835947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.836480 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.836702 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.836713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.838211 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.838215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.839216 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.839233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.840276 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.840282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.840482 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.841443 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.841462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.844171 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.848736 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.848740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.848744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.849304 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.850171 (+ 867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.850175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.850178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.850393 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.851043 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.851047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.851221 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.851367 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.851632 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 15:20:13.852319 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.835373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.2.246.124:41739)
0927 15:20:13.835602 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.835607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.835627 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:13.835678 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.835681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.835749 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.835873 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.836928 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.837930 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.839521 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.840022 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.840246 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.844513 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.844520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.844524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.844574 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.848463 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.849514 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.849520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.849523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.850004 (+ 481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.850536 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.850540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.850542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.850886 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.851015 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.851498 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.851823 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":98}
I20260927 15:20:13.857589 9936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Bootstrap starting.
I20260927 15:20:13.861472 9936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:13.862569 9936 log.cc:824] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:13.865716 9936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: No bootstrap required, opened a new log
I20260927 15:20:13.874042 9936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } }
I20260927 15:20:13.874447 9936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:13.874588 9936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276a84cd677549cbacf13a50293fdfb6, State: Initialized, Role: FOLLOWER
I20260927 15:20:13.875185 9936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } }
I20260927 15:20:13.877175 9943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } } }
I20260927 15:20:13.877631 9943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:13.877810 9936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:20:13.890017 9954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:13.890254 9954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:13.895078 9782 master_runner.cc:428] Detected that this master 276a84cd677549cbacf13a50293fdfb6 is joining an existing cluster
I20260927 15:20:13.895236 9782 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.246.124:41739
I20260927 15:20:13.913956 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9278
I20260927 15:20:13.924037 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.900247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.2.246.124:41739)
0927 15:20:13.900478 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.900487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.901811 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 15:20:13.902110 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.902118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.902210 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.902300 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.904002 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.906100 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.908651 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.909499 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.909814 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.914318 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.914326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.914332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.914373 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.918839 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.920022 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.920027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.920032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.920679 (+ 647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.921323 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.921331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.921337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.921799 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.922181 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.923009 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.923389 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":80}
I20260927 15:20:13.924887 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.900278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.0.0.1:56230)
0927 15:20:13.901662 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.901685 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.901935 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.902508 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.902517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.903290 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.903786 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.903804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.906393 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.906398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.908472 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.908487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.910178 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.910187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.910410 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.912469 (+ 2059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.912493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.913989 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.919036 (+ 5047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.919043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.919050 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.919862 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.920924 (+ 1062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.920932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.920940 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.921191 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.922322 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.922330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.922632 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.923975 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.924388 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":1,"thread_start_us":982,"threads_started":1}
I20260927 15:20:13.930408 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.900164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.2.246.126:35349)
0927 15:20:13.900358 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.900364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.902606 (+ 2242us) server_negotiation.cc:407] Connection header received
0927 15:20:13.903182 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.903189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.903302 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.903412 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.907488 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.908661 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.911389 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.912153 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.912281 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.919185 (+ 6904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.919194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.919202 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.919439 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.926078 (+ 6639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.927148 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.927155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.927162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.927346 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.927842 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.927850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.927856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.928229 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.928357 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.929570 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.930021 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:20:13.931912 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.901466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57246)
0927 15:20:13.902466 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.902491 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.902864 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.903699 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.903708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.906594 (+ 2886us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.907246 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.907260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.908845 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.908850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.911181 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.911196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.915805 (+ 4609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.915812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.916062 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.916814 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.916845 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.918880 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.926304 (+ 7424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.926311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.926315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.926977 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.927510 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.927515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.927519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.927707 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.928427 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.928431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.928874 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.930662 (+ 1788us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.931377 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":608,"threads_started":1}
I20260927 15:20:13.938304 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.902864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.2.246.125:37729)
0927 15:20:13.903214 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.903221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.908273 (+ 5052us) server_negotiation.cc:407] Connection header received
0927 15:20:13.908587 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.908592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.908679 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.908753 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.911047 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.912202 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.914555 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.915191 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.915297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.931189 (+ 15892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.931198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.931203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.931246 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.934895 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.935790 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.935795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.935798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.935959 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.936611 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.936615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.936618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.936967 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.937096 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.937684 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.937954 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 15:20:13.939344 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.903392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39746)
0927 15:20:13.907694 (+ 4302us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.907716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.908407 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.908994 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.908999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.909775 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.910845 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.910859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.912439 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.912445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.914341 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.914357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.922162 (+ 7805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.922169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.922793 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.925048 (+ 2255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.925076 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.930940 (+ 5864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.935072 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.935080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.935084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.935646 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.936105 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.936109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.936113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.936481 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.937174 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.937179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.937396 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.938566 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.938895 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3656,"mutex_wait_us":403,"thread_start_us":2639,"threads_started":1}
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.65@KRBTEST.COM
Sep 27 15:20:13 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522410, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.126@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
I20260927 15:20:13.954393 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.916083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.2.246.65:46669)
0927 15:20:13.930538 (+ 14455us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.930550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.930596 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:20:13.930696 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.930706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.931062 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.933510 (+ 2448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.933890 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.935616 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.938758 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.940068 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.940241 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.946254 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.946280 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.946305 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.946367 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.950966 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.951834 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.951841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.951844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.952021 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.952418 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.952423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.952426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.952844 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.953170 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.953711 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.954041 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14196,"thread_start_us":14006,"threads_started":1}
I20260927 15:20:13.956166 9401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:13.964915 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.944472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.0.0.1:56236)
0927 15:20:13.944675 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:13.944707 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:13.944858 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:13.945377 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:13.945385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:13.945916 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:13.946155 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.946174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.947366 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.947374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.948917 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:13.948935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.950234 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.950242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:13.950497 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.951229 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:13.951255 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:13.955267 (+ 4012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:13.960365 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.960375 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.960407 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.961112 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.961736 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:13.961742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:13.961746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:13.961965 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:13.963770 (+ 1805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:13.963776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:13.963977 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 15:20:13.964158 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 15:20:13.964526 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:20:13.965164 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:13.944673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.2.246.124:41739)
0927 15:20:13.944873 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:13.944879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:13.944903 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:20:13.945022 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:13.945028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:13.945159 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:13.945280 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:13.946331 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.947186 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.949100 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:13.950001 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:13.950172 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:13.955538 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:13.955546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:13.955551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:13.955604 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:13.960116 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.961289 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.961298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.961304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.961534 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:13.962121 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:13.962132 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:13.962137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:13.963545 (+ 1408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:13.963798 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:13.964376 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 15:20:13.964711 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:20:13.967341 9217 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.246.124:41739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "276a84cd677549cbacf13a50293fdfb6"
member_type: NON_VOTER
last_known_addr {
host: "127.2.246.124"
port: 41739
}
attrs {
promote: true
}
}
dest_uuid: "01abce2181f74267bccc54e24211dd8d"
cas_config_opid_index: 5
I20260927 15:20:13.968582 9217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } }
I20260927 15:20:13.971104 9877 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:13.971992 9827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:20:13.974967 9911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:13.979036 9911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:20:13.984261 9965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 15:20:13.986783 9965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:13.989423 9827 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:13.994069 9855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: NON_VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: true } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:13.994439 9855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: LEARNER
W20260927 15:20:13.997087 9890 tablet.cc:2369] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:13.998613 9877 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.246.124:41739
I20260927 15:20:13.999481 9782 master.cc:537] Master@127.2.246.124:41739 shutting down...
I20260927 15:20:14.003068 9877 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: attempting to promote NON_VOTER 1503936f32ae4a87ac1c183c9a1b0308 to VOTER
I20260927 15:20:14.007910 9877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } }
I20260927 15:20:14.010715 9827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 LEARNER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:20:14.011686 9965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 15:20:14.015209 9965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.018957 9827 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
W20260927 15:20:14.022387 9203 consensus_peers.cc:487] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d -> Peer 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124:41739): Couldn't send request to peer 276a84cd677549cbacf13a50293fdfb6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:20:14.027026 9855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.027376 9855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:14.028535 9965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.028976 9965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:14.028797 9967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.029238 9967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:14.029522 9966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.029917 9966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:14.036605 9782 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:14.037242 9782 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:14.037508 9782 tablet_replica.cc:330] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: stopping tablet replica
I20260927 15:20:14.046212 9782 master.cc:552] Master@127.2.246.124:41739 shutdown complete.
I20260927 15:20:14.046401 9782 master_runner.cc:305] Clearing existing system tablet
I20260927 15:20:14.046867 9782 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.047062 9782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.050588 9782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 15:20:14.052762 9973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.053334 9782 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.053803 9782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 15:20:14.053994 9782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "276a84cd677549cbacf13a50293fdfb6"
format_stamp: "Formatted at 2026-09-27 15:20:13 on dist-test-slave-14jq"
server_key: "1c58c35a76ba319425b591cf110a7980"
server_key_iv: "ab86289d7f525e3274226e0405f75a8b"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.059280 9782 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:14.067413 9782 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:20:14.067746 9782 log.cc:1196] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:20:14.068518 9782 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Deleting consensus metadata
I20260927 15:20:14.068873 9782 master_runner.cc:315] Copying system tablet from 127.2.246.126:35349
I20260927 15:20:14.072268 9782 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.246.126:35349
I20260927 15:20:14.090142 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9278
I20260927 15:20:14.091059 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.073403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.2.246.126:35349)
0927 15:20:14.073636 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.073643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.074261 (+ 618us) server_negotiation.cc:407] Connection header received
0927 15:20:14.074567 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.074573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.074652 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.074723 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:14.075990 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.077109 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.079392 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.080040 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.080167 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.083006 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:14.083013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:14.083018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:14.083064 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:14.087261 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.088275 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.088283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.088290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.088457 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.088927 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.088932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.088935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.089296 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:14.089455 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.090092 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.090252 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 15:20:14.092284 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.073395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57258)
0927 15:20:14.074125 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.074148 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.074397 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.074938 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:14.074943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:14.075524 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:14.075811 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.075824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.077291 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.077298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.079134 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.079159 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.080248 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.080255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.080469 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.081353 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:14.081381 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:14.082722 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:14.087570 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.087579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.087585 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.088115 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.088611 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.088616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.088623 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.088797 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.089554 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:14.089559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:14.089758 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 15:20:14.091302 (+ 1544us) client_negotiation.cc:240] Negotiation successful
0927 15:20:14.091722 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260927 15:20:14.093034 9237 tablet_copy_service.cc:140] P 01abce2181f74267bccc54e24211dd8d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 276a84cd677549cbacf13a50293fdfb6 ({username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:57258)
I20260927 15:20:14.093185 9237 tablet_copy_service.cc:161] P 01abce2181f74267bccc54e24211dd8d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 276a84cd677549cbacf13a50293fdfb6 at {username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:57258: session id = 276a84cd677549cbacf13a50293fdfb6-00000000000000000000000000000000
I20260927 15:20:14.097589 9237 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:20:14.099923 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:20:14.108319 9782 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: tablet copy: Starting download of 0 data blocks...
I20260927 15:20:14.108749 9782 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: tablet copy: Starting download of 1 WAL segments...
I20260927 15:20:14.112922 9782 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:20:14.114602 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:46669
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:35349
--webserver_port=46025
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:14.119724 9237 tablet_copy_service.cc:342] P 01abce2181f74267bccc54e24211dd8d: Request end of tablet copy session 276a84cd677549cbacf13a50293fdfb6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:57258
I20260927 15:20:14.119942 9237 tablet_copy_service.cc:433] P 01abce2181f74267bccc54e24211dd8d: ending tablet copy session 276a84cd677549cbacf13a50293fdfb6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 276a84cd677549cbacf13a50293fdfb6
I20260927 15:20:14.138150 9782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:14.154251 9981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.158129 9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:14.170183 9782 server_base.cc:1034] running on GCE node
W20260927 15:20:14.172299 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:14.173020 9782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:14.174191 9782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:14.175427 9782 hybrid_clock.cc:648] HybridClock initialized: now 1790522414175368 us; error 64 us; skew 500 ppm
I20260927 15:20:14.175741 9782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:14.178658 9782 init.cc:377] Logged in from keytab as oryx/127.2.246.124@KRBTEST.COM (short username oryx)
I20260927 15:20:14.183308 9782 webserver.cc:460] Webserver started at http://127.2.246.124:39115/ using document root <none> and password file <none>
I20260927 15:20:14.183806 9782 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.183923 9782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.187774 9782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:20:14.190353 9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.190963 9782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:14.191152 9782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "276a84cd677549cbacf13a50293fdfb6"
format_stamp: "Formatted at 2026-09-27 15:20:13 on dist-test-slave-14jq"
server_key: "1c58c35a76ba319425b591cf110a7980"
server_key_iv: "ab86289d7f525e3274226e0405f75a8b"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.191380 9782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.235347 9782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:14.240787 9782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:14.309217 9782 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.124:41739
I20260927 15:20:14.309370 10042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.124:41739 every 8 connection(s)
I20260927 15:20:14.310637 9782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
I20260927 15:20:14.317971 10043 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:20:14.322536 10043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Bootstrap starting.
I20260927 15:20:14.345038 10043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:20:14.345598 10043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Bootstrap complete.
I20260927 15:20:14.347008 10043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } }
I20260927 15:20:14.347307 10043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 276a84cd677549cbacf13a50293fdfb6, State: Initialized, Role: LEARNER
I20260927 15:20:14.347656 10043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } }
I20260927 15:20:14.364926 10043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:14.364521 10046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.365422 10046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:20:14.378858 10057 catalog_manager.cc:1259] Loaded cluster ID: bfa39b6ae0a64801b3a56667a890e0de
I20260927 15:20:14.379057 10057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: loading cluster ID for follower catalog manager: success
I20260927 15:20:14.386241 10057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: acquiring CA information for follower catalog manager: success
I20260927 15:20:14.390509 10057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:20:14.416728 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.405424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.2.246.124:41739)
0927 15:20:14.406410 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.406418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.406446 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:20:14.406546 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.406551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.406691 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.406870 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:14.407587 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.409409 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.412306 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.414817 (+ 2511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.415097 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.415542 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.416112 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.416326 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":579,"threads_started":1}
I20260927 15:20:14.416772 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.405114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.0.0.1:56252)
0927 15:20:14.405489 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.405516 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.405721 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.406996 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:14.407002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:14.407052 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:14.407319 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.407344 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.409904 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.409911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.411921 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.411955 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.415207 (+ 3252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.415219 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.415742 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.415818 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 15:20:14.416018 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 15:20:14.416290 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260927 15:20:14.418846 10018 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 15:20:14.419653 10046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01abce2181f74267bccc54e24211dd8d. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:14.420063 10046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: LEARNER
W20260927 15:20:14.555560 9980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:14.555975 9980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:14.556128 9980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:14.565342 9980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:14.565493 9980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:14.565622 9980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:14.577826 9980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:46669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=46025
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:14.579403 9980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:14.580814 9980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:14.598223 10063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.598353 10066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.598353 10064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:14.599843 9980 server_base.cc:1034] running on GCE node
I20260927 15:20:14.600849 9980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:14.602295 9980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:14.603598 9980 hybrid_clock.cc:648] HybridClock initialized: now 1790522414603543 us; error 51 us; skew 500 ppm
I20260927 15:20:14.604048 9980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:14.608443 9980 init.cc:377] Logged in from keytab as oryx/127.2.246.65@KRBTEST.COM (short username oryx)
I20260927 15:20:14.610603 9980 webserver.cc:460] Webserver started at http://127.2.246.65:46025/ using document root <none> and password file <none>
I20260927 15:20:14.611644 9980 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.611855 9980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.622218 9980 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:20:14.626971 10073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.629105 9980 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:20:14.630213 9980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c11e7745dc864f68b0cad8fefdf23540"
format_stamp: "Formatted at 2026-09-27 15:20:10 on dist-test-slave-14jq"
server_key: "67cbb7701583ae7b8d51d4cb44310336"
server_key_iv: "71b9f73e0d3182ef7c438634af7a60d6"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.631466 9980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.661392 9980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:14.667778 9980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:14.668160 9980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:14.669736 9980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:14.672760 9980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:14.672912 9980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.673044 9980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:14.673142 9980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.720100 9980 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:46669
I20260927 15:20:14.720201 10185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:46669 every 8 connection(s)
I20260927 15:20:14.722137 9980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:14.732414 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 9980
I20260927 15:20:14.732785 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9412
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
I20260927 15:20:14.757128 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.725774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:44083 (local address 127.2.246.124:41739)
0927 15:20:14.726019 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.726025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.727810 (+ 1785us) server_negotiation.cc:407] Connection header received
0927 15:20:14.731438 (+ 3628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.731445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.731551 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.731631 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:14.734988 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.736206 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.740024 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.740882 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.741198 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.748728 (+ 7530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:14.748737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:14.748742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:14.748786 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:14.752484 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.753980 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.753987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.753992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.754159 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754767 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.754772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.754775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.755163 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:14.755293 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.756500 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.756770 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:20:14.757548 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.725861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:52969 (local address 127.2.246.125:37729)
0927 15:20:14.726398 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.726404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.727911 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 15:20:14.731396 (+ 3485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.731402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.731522 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.731645 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:14.735946 (+ 4301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.737653 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.740676 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.741595 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.741723 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.747978 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:14.747986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:14.747990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:14.748057 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:14.752614 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754005 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.754011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.754016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.754180 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.755544 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.755551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.755557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.755973 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:14.756124 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.756807 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.757082 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":278,"threads_started":1}
Sep 27 15:20:14 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.66@KRBTEST.COM
I20260927 15:20:14.758376 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.728909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:59715 (local address 127.2.246.126:35349)
0927 15:20:14.729523 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.729527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.729545 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:20:14.731550 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.731557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.731645 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.731765 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:14.735387 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.736462 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.740021 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.740901 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.741180 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.747965 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:14.747973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:14.747979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:14.748036 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:14.752594 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754005 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.754011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.754016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.754189 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.755601 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.755609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.755615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.755973 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:14.756116 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.756815 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.757082 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":404,"threads_started":1}
I20260927 15:20:14.759912 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.728640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:59715)
0927 15:20:14.729334 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.729360 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.731064 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.731974 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:14.731979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:14.733520 (+ 1541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:14.735199 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.735210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.736646 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.736650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.739750 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.739786 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.741606 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.741615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.742147 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.743384 (+ 1237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:14.743430 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:14.747642 (+ 4212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:14.752855 (+ 5213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.752861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.752866 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.753796 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.754343 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.754352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.755472 (+ 1120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.756203 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:14.756208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:14.756470 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 15:20:14.758919 (+ 2449us) client_negotiation.cc:240] Negotiation successful
0927 15:20:14.759250 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":446,"spinlock_wait_cycles":23296,"thread_start_us":363,"threads_started":1}
I20260927 15:20:14.759881 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.727091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.2.246.65:52969)
0927 15:20:14.727776 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.727802 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.731070 (+ 3268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.732007 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:14.732015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:14.735082 (+ 3067us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:14.735666 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.735687 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.737960 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.737967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.740425 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.740448 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.742174 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.742183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.742509 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.744197 (+ 1688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:14.744217 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:14.747160 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:14.752860 (+ 5700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.752865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.752868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.753796 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.754343 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.754352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.755387 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.756203 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:14.756208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:14.756470 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 15:20:14.757910 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0927 15:20:14.758428 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":53,"spinlock_wait_cycles":29184,"thread_start_us":1131,"threads_started":1}
I20260927 15:20:14.759965 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.726544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.2.246.65:44083)
0927 15:20:14.727529 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.727592 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.731064 (+ 3472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.731964 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:14.731979 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:14.733520 (+ 1541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:14.734782 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.734815 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.736397 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.736403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.739777 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:14.739786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.741640 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.741646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:14.742147 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.743384 (+ 1237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:14.743431 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:14.748445 (+ 5014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:14.752668 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.752678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.752697 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.753790 (+ 1093us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.754308 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:14.754316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:14.754321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:14.754622 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:14.755369 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:14.755430 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:14.756013 (+ 583us) client_negotiation.cc:769] Sending connection context
0927 15:20:14.757370 (+ 1357us) client_negotiation.cc:240] Negotiation successful
0927 15:20:14.758048 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":463,"spinlock_wait_cycles":73600,"thread_start_us":639,"threads_started":1}
I20260927 15:20:14.762113 10187 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:14.762531 10188 heartbeater.cc:347] Connected to a master server at 127.2.246.125:37729
I20260927 15:20:14.762573 10187 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:14.762755 10188 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:14.762787 10186 heartbeater.cc:347] Connected to a master server at 127.2.246.124:41739
I20260927 15:20:14.762889 10186 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:14.764022 10186 heartbeater.cc:511] Master 127.2.246.124:41739 requested a full tablet report, sending...
I20260927 15:20:14.764070 10188 heartbeater.cc:511] Master 127.2.246.125:37729 requested a full tablet report, sending...
I20260927 15:20:14.764106 10187 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:14.766685 9217 ts_manager.cc:194] Re-registered known tserver with Master: c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
I20260927 15:20:14.767221 9817 ts_manager.cc:194] Registered new tserver with Master: c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
I20260927 15:20:14.767199 10008 ts_manager.cc:194] Registered new tserver with Master: c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
I20260927 15:20:14.767865 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:59715
I20260927 15:20:14.769819 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.733933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.2.246.66:43861)
0927 15:20:14.734853 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:14.734863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:14.744195 (+ 9332us) server_negotiation.cc:407] Connection header received
0927 15:20:14.744338 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:14.744350 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:14.744784 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:14.745027 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:14.747653 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.749269 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.751776 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:14.752864 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:14.753001 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:14.759364 (+ 6363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:14.759380 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:14.759397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:14.759437 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:14.764028 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.765210 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.765221 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.765225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.765528 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:14.766126 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:14.766132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:14.766136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:14.766624 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:14.767656 (+ 1032us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:14.767892 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:20:14.768948 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":509,"threads_started":1}
I20260927 15:20:14.772226 9535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:14.860514 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9412
I20260927 15:20:14.879267 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:43861
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:35349
--webserver_port=46513
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:15.012184 10199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: NON_VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: true } } }
I20260927 15:20:15.012739 10199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:15.013970 10199 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: attempting to promote NON_VOTER 276a84cd677549cbacf13a50293fdfb6 to VOTER
I20260927 15:20:15.014662 10199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:15.017251 9827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:20:15.017750 10018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 LEARNER]: Refusing update from remote peer 01abce2181f74267bccc54e24211dd8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:20:15.018435 10201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:20:15.019038 10200 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:20:15.022924 10200 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.024008 9827 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.027559 10018 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.034127 10204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.034899 10204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:15.035108 10199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.036207 10199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:15.036998 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.037438 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:15.040584 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.041306 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:15.042531 10200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01abce2181f74267bccc54e24211dd8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:15.042820 10200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d [sys.catalog]: This master's current role is: LEADER
W20260927 15:20:15.258714 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:15.259084 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:15.259275 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:15.268209 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:15.268379 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:15.268586 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:15.281075 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:43861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=46513
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:15.282698 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:15.284260 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:15.303061 10212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:15.304699 10198 server_base.cc:1034] running on GCE node
W20260927 15:20:15.306926 10215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:15.306877 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:15.308039 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:15.309206 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:15.312606 10198 hybrid_clock.cc:648] HybridClock initialized: now 1790522415312508 us; error 83 us; skew 500 ppm
I20260927 15:20:15.313035 10198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:15.316536 10198 init.cc:377] Logged in from keytab as oryx/127.2.246.66@KRBTEST.COM (short username oryx)
I20260927 15:20:15.318332 10198 webserver.cc:460] Webserver started at http://127.2.246.66:46513/ using document root <none> and password file <none>
I20260927 15:20:15.319043 10198 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:15.319160 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:15.325142 10198 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:20:15.328555 10222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:15.330320 10198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:20:15.330576 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "b33c24343375466f8861a09c8e6dc5d1"
format_stamp: "Formatted at 2026-09-27 15:20:11 on dist-test-slave-14jq"
server_key: "00b1e0932fa09a69f4ad657b785e2de7"
server_key_iv: "b0ec4fafdb14afbeb15129f19fe605ed"
server_key_version: "encryptionkey@0"
I20260927 15:20:15.331665 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:15.355526 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:15.360710 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:15.360975 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:15.362174 10198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:15.363857 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:15.364003 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:15.364166 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:15.364264 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:15.407981 10198 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:43861
I20260927 15:20:15.408095 10334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:43861 every 8 connection(s)
I20260927 15:20:15.409518 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:15.417411 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 10198
I20260927 15:20:15.417793 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9546
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.67@KRBTEST.COM
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:15.460043 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.419277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.2.246.67:35785)
0927 15:20:15.424848 (+ 5571us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.424875 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.430983 (+ 6108us) server_negotiation.cc:407] Connection header received
0927 15:20:15.431319 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.431337 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.431679 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.431837 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.435179 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.437289 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.439945 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.441093 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.441262 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.449466 (+ 8204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.449492 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.449518 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.449569 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.455152 (+ 5583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.456321 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.456332 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.456336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.456541 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.457117 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.457123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.457129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.457622 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.457987 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.459244 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.459543 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5299,"thread_start_us":5072,"threads_started":1}
I20260927 15:20:15.461764 9669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:15.468061 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.413687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36493 (local address 127.2.246.124:41739)
0927 15:20:15.426845 (+ 13158us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.426854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.435044 (+ 8190us) server_negotiation.cc:407] Connection header received
0927 15:20:15.439083 (+ 4039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.439089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.439187 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.439281 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.443054 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.444499 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.448366 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.449165 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.449297 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.457703 (+ 8406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.457714 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.457719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.457773 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.464277 (+ 6504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.465312 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.465319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.465322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.465547 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.466046 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.466051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.466055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.466425 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.466555 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.467404 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.467668 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12987,"thread_start_us":13170,"threads_started":1}
I20260927 15:20:15.468264 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.413738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36325 (local address 127.2.246.126:35349)
0927 15:20:15.430342 (+ 16604us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.430350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.438642 (+ 8292us) server_negotiation.cc:407] Connection header received
0927 15:20:15.438846 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.438852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.438967 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.439085 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.443035 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.444188 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.448380 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.449076 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.449172 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.458066 (+ 8894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.458074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.458079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.458139 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.463107 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.464372 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.464383 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.464388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.464602 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.465307 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.465313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.465316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.465780 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.466026 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.467430 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.467841 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16436,"thread_start_us":16311,"threads_started":1}
I20260927 15:20:15.469594 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.432793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:39185 (local address 127.2.246.125:37729)
0927 15:20:15.442765 (+ 9972us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.442770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.442786 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:20:15.442839 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.442842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.442898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.443034 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.444354 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.445326 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.448208 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.448867 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.448994 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.456177 (+ 7183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.456185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.456190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.456246 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.459894 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.461015 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.461021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.461024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.461184 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.465435 (+ 4251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.465442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.465445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.465926 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.466185 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.468870 (+ 2685us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.469153 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9868,"thread_start_us":9741,"threads_started":1}
I20260927 15:20:15.470276 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.414740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:36325)
0927 15:20:15.438469 (+ 23729us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.438494 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:15.438703 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:15.439702 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:15.439719 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:15.442444 (+ 2725us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:15.442812 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.442830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.444395 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.444401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.448224 (+ 3823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.448236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.449524 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.449529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.449697 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.450784 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:15.450826 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:15.457744 (+ 6918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:15.463418 (+ 5674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.463425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.463430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.464149 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.464779 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.464785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.464789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.465144 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.466146 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:15.466157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:15.466914 (+ 757us) client_negotiation.cc:769] Sending connection context
0927 15:20:15.468660 (+ 1746us) client_negotiation.cc:240] Negotiation successful
0927 15:20:15.469118 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23501,"mutex_wait_us":728,"spinlock_wait_cycles":92160,"thread_start_us":16316,"threads_started":1}
I20260927 15:20:15.470233 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.414740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.2.246.66:36493)
0927 15:20:15.432141 (+ 17401us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.432257 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:15.437329 (+ 5072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:15.439703 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:15.439719 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:15.440675 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:15.441632 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.441664 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.444736 (+ 3072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.444744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.448122 (+ 3378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.448147 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.449768 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.449775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.450162 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.451049 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:15.451083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:15.457307 (+ 6224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:15.464476 (+ 7169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.464483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.464490 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.465123 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.465671 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.465679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.465686 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.465914 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.466637 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:15.466642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:15.466919 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 15:20:15.468349 (+ 1430us) client_negotiation.cc:240] Negotiation successful
0927 15:20:15.468717 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16838,"mutex_wait_us":366,"spinlock_wait_cycles":7552,"thread_start_us":23094,"threads_started":1}
I20260927 15:20:15.470676 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.439061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.2.246.66:39185)
0927 15:20:15.442084 (+ 3023us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.442111 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:15.442283 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:15.443224 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:15.443229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:15.443879 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:15.444195 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.444205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.445506 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.445513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.448039 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.448054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.449308 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.449318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.449687 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.450784 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:15.450818 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:15.455850 (+ 5032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:15.460122 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.460134 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.460159 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.460861 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.461321 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.461325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.461332 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.465145 (+ 3813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.466248 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:15.466253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:15.466914 (+ 661us) client_negotiation.cc:769] Sending connection context
0927 15:20:15.467285 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 15:20:15.467992 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2806,"mutex_wait_us":194,"spinlock_wait_cycles":27392,"thread_start_us":2604,"threads_started":1}
I20260927 15:20:15.472651 10336 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:15.473362 10336 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:15.474121 10337 heartbeater.cc:347] Connected to a master server at 127.2.246.125:37729
I20260927 15:20:15.474165 10335 heartbeater.cc:347] Connected to a master server at 127.2.246.124:41739
I20260927 15:20:15.474287 10337 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:15.474395 10335 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:15.475960 10337 heartbeater.cc:511] Master 127.2.246.125:37729 requested a full tablet report, sending...
I20260927 15:20:15.476058 10335 heartbeater.cc:511] Master 127.2.246.124:41739 requested a full tablet report, sending...
I20260927 15:20:15.475991 10336 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:15.478631 10008 ts_manager.cc:194] Registered new tserver with Master: b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:15.478986 9217 ts_manager.cc:194] Re-registered known tserver with Master: b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:15.479223 9817 ts_manager.cc:194] Registered new tserver with Master: b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:15.480238 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:36325
I20260927 15:20:15.553081 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9546
I20260927 15:20:15.571923 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:35785
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:35349
--webserver_port=35257
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:15.771921 10187 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
W20260927 15:20:15.919155 10348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:15.919453 10348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:15.919553 10348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:15.927047 10348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:15.927184 10348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:15.927299 10348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:15.937708 10348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:35785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=35257
--tserver_master_addrs=127.2.246.126:35349,127.2.246.125:37729,127.2.246.124:41739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:15.939105 10348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:15.940368 10348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:15.956015 10355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:15.956060 10357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:15.956512 10354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:15.957315 10348 server_base.cc:1034] running on GCE node
I20260927 15:20:15.958374 10348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:15.959735 10348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:15.961117 10348 hybrid_clock.cc:648] HybridClock initialized: now 1790522415961064 us; error 46 us; skew 500 ppm
I20260927 15:20:15.961596 10348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:15 dist-test-slave-14jq krb5kdc[9172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:15.969525 10348 init.cc:377] Logged in from keytab as oryx/127.2.246.67@KRBTEST.COM (short username oryx)
I20260927 15:20:15.971745 10348 webserver.cc:460] Webserver started at http://127.2.246.67:35257/ using document root <none> and password file <none>
I20260927 15:20:15.972699 10348 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:15.972869 10348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:15.979378 10348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:20:15.982836 10364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:15.984551 10348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:15.984766 10348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "c607a92dcead4686badf99847336fcd8"
format_stamp: "Formatted at 2026-09-27 15:20:12 on dist-test-slave-14jq"
server_key: "dc600d168b7a30aed80fb98fa02ac9a1"
server_key_iv: "124cca41675fd1ef27fd41f4a904ed1f"
server_key_version: "encryptionkey@0"
I20260927 15:20:15.985826 10348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:16.004559 10348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:16.009258 10348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:16.009567 10348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:16.010707 10348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:16.012283 10348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:16.012413 10348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:16.012537 10348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:16.012634 10348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:16.067497 10348 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:35785
I20260927 15:20:16.067555 10476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:35785 every 8 connection(s)
I20260927 15:20:16.069733 10348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:16.078558 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 10348
Sep 27 15:20:16 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.125@KRBTEST.COM
I20260927 15:20:16.125975 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.077545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:59601 (local address 127.2.246.125:37729)
0927 15:20:16.078192 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.078200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.092269 (+ 14069us) server_negotiation.cc:407] Connection header received
0927 15:20:16.102105 (+ 9836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.102114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.102217 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.102313 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.104158 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.105673 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.109135 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.110205 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.110429 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.115720 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.115732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.115738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.115806 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.120124 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.121495 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.121501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.121504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.122084 (+ 580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.122769 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.122776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.122781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.123199 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.123404 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.124372 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.124693 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":312,"threads_started":1}
I20260927 15:20:16.126241 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.078055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.2.246.67:59601)
0927 15:20:16.092116 (+ 14061us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.092137 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.101869 (+ 9732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.102625 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.102633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.103513 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:16.103933 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.103951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.106005 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.106012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.108894 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.108911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.110519 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.110526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.110863 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.111863 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:16.111894 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:16.115363 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:16.120308 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.120324 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.120353 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.121149 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.122246 (+ 1097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.122252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.122255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.122545 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.123386 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:16.123395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:16.123867 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.124201 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.124775 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13918,"thread_start_us":13668,"threads_started":1}
I20260927 15:20:16.128257 10479 heartbeater.cc:347] Connected to a master server at 127.2.246.125:37729
I20260927 15:20:16.128713 10479 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:16.129937 10479 heartbeater.cc:511] Master 127.2.246.125:37729 requested a full tablet report, sending...
Sep 27 15:20:16 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
Sep 27 15:20:16 dist-test-slave-14jq krb5kdc[9172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522415, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.124@KRBTEST.COM
I20260927 15:20:16.134961 9817 ts_manager.cc:194] Registered new tserver with Master: c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785)
I20260927 15:20:16.151964 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.074074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:53881 (local address 127.2.246.126:35349)
0927 15:20:16.074825 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.074833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.076628 (+ 1795us) server_negotiation.cc:407] Connection header received
0927 15:20:16.096150 (+ 19522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.096161 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.096290 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.096385 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.112305 (+ 15920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.113422 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.125576 (+ 12154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.126207 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.126326 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.136829 (+ 10503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.137246 (+ 417us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.137258 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.137317 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.142497 (+ 5180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.146452 (+ 3955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.146459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.146463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.146641 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.149809 (+ 3168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.149816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.149819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.150360 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.150518 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.151288 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.151544 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":384,"threads_started":1}
I20260927 15:20:16.152925 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.075203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:53881)
0927 15:20:16.076364 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.076432 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.095620 (+ 19188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.097093 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.097110 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.098098 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:16.099084 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.099114 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.113700 (+ 14586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.113708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.115826 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.115840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.127346 (+ 11506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.127354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.127595 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.129648 (+ 2053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:16.129685 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:16.136511 (+ 6826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:16.144734 (+ 8223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.144743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.144750 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.145424 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.147809 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.147818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.147825 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.149664 (+ 1839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.150739 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:16.150747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:16.150935 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.151094 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.151504 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":582,"threads_started":1}
I20260927 15:20:16.154368 10478 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35349
I20260927 15:20:16.154548 10478 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:16.155169 10478 heartbeater.cc:511] Master 127.2.246.126:35349 requested a full tablet report, sending...
I20260927 15:20:16.156401 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.083309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.2.246.126:35349)
0927 15:20:16.103573 (+ 20264us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.103581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.103694 (+ 113us) server_negotiation.cc:407] Connection header received
0927 15:20:16.103800 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.103810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.103935 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.104223 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.121520 (+ 17297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.123208 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.134851 (+ 11643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.135655 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.135776 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.144195 (+ 8419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.144202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.144207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.144267 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.149111 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.151478 (+ 2367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.151488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.151494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.151703 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.153780 (+ 2077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.153788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.153794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.154297 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.154592 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.155455 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.155774 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20058,"thread_start_us":21139,"threads_started":1}
I20260927 15:20:16.156476 9217 ts_manager.cc:194] Re-registered known tserver with Master: c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785)
I20260927 15:20:16.158282 9217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:53881
I20260927 15:20:16.158530 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.107341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.2.246.124:41739)
0927 15:20:16.111080 (+ 3739us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.111086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.111117 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:20:16.111178 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.111183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.111262 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.111799 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.116875 (+ 5076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.118177 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.120329 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.120952 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.121050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.144888 (+ 23838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.144897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.144904 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.144957 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.151238 (+ 6281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.153385 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.153394 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.153401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.153590 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.154141 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.154147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.154151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.156156 (+ 2005us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.156419 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.157624 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.157969 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3594,"thread_start_us":4377,"threads_started":1}
I20260927 15:20:16.159927 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.083125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.2.246.67:48167)
0927 15:20:16.109009 (+ 25884us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.109032 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.109156 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.116506 (+ 7350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.116513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.117167 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:16.117522 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.117536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.121482 (+ 3946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.121489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.124399 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.124411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.130779 (+ 6368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.130787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.131065 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.132253 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:16.132283 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:16.137637 (+ 5354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:16.145600 (+ 7963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.145606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.145613 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.146224 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.156983 (+ 10759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:16.156989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:16.156993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:16.157218 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:16.158708 (+ 1490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:16.158717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:16.158928 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.159084 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.159488 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25723,"thread_start_us":25502,"threads_started":1}
I20260927 15:20:16.163810 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.083994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:48167 (local address 127.2.246.124:41739)
0927 15:20:16.107349 (+ 23355us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.107356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.109272 (+ 1916us) server_negotiation.cc:407] Connection header received
0927 15:20:16.109356 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.109364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.109464 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.109618 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.117757 (+ 8139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.119396 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.124578 (+ 5182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.125199 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.125334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.137959 (+ 12625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.137969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.137976 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.138030 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.143216 (+ 5186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.149212 (+ 5996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.149221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.149225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.149422 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.157637 (+ 8215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.157642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.157645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.158459 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.158631 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.160338 (+ 1707us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.160604 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23209,"thread_start_us":23086,"threads_started":1}
I20260927 15:20:16.165452 10477 heartbeater.cc:347] Connected to a master server at 127.2.246.124:41739
I20260927 15:20:16.165621 10477 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:16.165810 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.101162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.2.246.125:37729)
0927 15:20:16.122456 (+ 21294us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.122462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.122481 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:20:16.122546 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.122551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.122640 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.122794 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.127984 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.129342 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.134062 (+ 4720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.134971 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.135148 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.143603 (+ 8455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.143613 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.143621 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.143676 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.150669 (+ 6993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.154881 (+ 4212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.154890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.154896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.155115 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.160941 (+ 5826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.160951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.160957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.161491 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.161801 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.164906 (+ 3105us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.165333 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":21149,"thread_start_us":21871,"threads_started":1}
I20260927 15:20:16.166196 10477 heartbeater.cc:511] Master 127.2.246.124:41739 requested a full tablet report, sending...
I20260927 15:20:16.168027 10008 ts_manager.cc:194] Registered new tserver with Master: c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785)
I20260927 15:20:16.165200 9217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:16.181033 9217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:16.271836 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.225120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35785 (local address 127.0.0.1:44834)
0927 15:20:16.260087 (+ 34967us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.260111 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.260275 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.261400 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.261404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.261428 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:16.263193 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.263214 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.265721 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.265731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.268073 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.268094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.270534 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.270544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.270840 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.270900 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.271033 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.271246 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34786,"mutex_wait_us":3,"thread_start_us":34360,"threads_started":1}
I20260927 15:20:16.271946 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.225828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.2.246.67:35785)
0927 15:20:16.258520 (+ 32692us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.258532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.260299 (+ 1767us) server_negotiation.cc:407] Connection header received
0927 15:20:16.260506 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.260521 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.261028 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.261386 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:16.263546 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.265486 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.268287 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.270255 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.270407 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.270694 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.271161 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.271431 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32370,"thread_start_us":32669,"threads_started":1}
I20260927 15:20:16.285102 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.226331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43861 (local address 127.0.0.1:49776)
0927 15:20:16.263828 (+ 37497us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.263856 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.264143 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.267260 (+ 3117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.267269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.267311 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:16.268159 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.268180 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.279698 (+ 11518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.279704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.281623 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.281641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.283841 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.283849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.284047 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.284081 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.284175 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.284321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37308,"thread_start_us":36566,"threads_started":1}
I20260927 15:20:16.285923 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.227534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.2.246.66:43861)
0927 15:20:16.257415 (+ 29881us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.257433 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.264000 (+ 6567us) server_negotiation.cc:407] Connection header received
0927 15:20:16.266195 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.266214 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.266645 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.266907 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:16.278002 (+ 11095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.279474 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.281853 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.283292 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.283466 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.283690 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.285427 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.285632 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29552,"thread_start_us":29924,"threads_started":1}
I20260927 15:20:16.288882 10412 tablet_service.cc:1467] Processing CreateTablet for tablet 5c21c45337c64389bc02782200ec2a2f (DEFAULT_TABLE table=test-table [id=3b33e498600c4bbba859d8f979b6790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:16.292397 10412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c21c45337c64389bc02782200ec2a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:16.294335 10270 tablet_service.cc:1467] Processing CreateTablet for tablet 5c21c45337c64389bc02782200ec2a2f (DEFAULT_TABLE table=test-table [id=3b33e498600c4bbba859d8f979b6790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:16.296379 10270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c21c45337c64389bc02782200ec2a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:16.307551 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.260572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.2.246.65:46669)
0927 15:20:16.261172 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.261180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.269648 (+ 8468us) server_negotiation.cc:407] Connection header received
0927 15:20:16.270181 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.270194 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.270582 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.270922 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:16.272207 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.291216 (+ 19009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.302233 (+ 11017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.304301 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.304494 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.304802 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.306769 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.307104 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":331,"threads_started":1}
I20260927 15:20:16.309805 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.261227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46669 (local address 127.0.0.1:33866)
0927 15:20:16.268945 (+ 7718us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.268971 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.269876 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.271281 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.271286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.271315 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:16.271834 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.271851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.296735 (+ 24884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.296744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.299178 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.299208 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.305333 (+ 6125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.305345 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.306008 (+ 663us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.306274 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.307914 (+ 1640us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.308522 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7468,"thread_start_us":7237,"threads_started":1}
I20260927 15:20:16.324923 10514 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Bootstrap starting.
I20260927 15:20:16.326817 10121 tablet_service.cc:1467] Processing CreateTablet for tablet 5c21c45337c64389bc02782200ec2a2f (DEFAULT_TABLE table=test-table [id=3b33e498600c4bbba859d8f979b6790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:16.328761 10121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c21c45337c64389bc02782200ec2a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:16.333698 10515 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Bootstrap starting.
I20260927 15:20:16.335698 10514 tablet_bootstrap.cc:654] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:16.342082 10515 tablet_bootstrap.cc:654] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:16.343816 10514 log.cc:824] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:16.346104 10515 log.cc:824] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:16.365252 10515 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: No bootstrap required, opened a new log
I20260927 15:20:16.365607 10515 ts_tablet_manager.cc:1397] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.016s
I20260927 15:20:16.368480 10514 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: No bootstrap required, opened a new log
I20260927 15:20:16.368853 10514 ts_tablet_manager.cc:1397] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.010s
I20260927 15:20:16.374770 10518 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Bootstrap starting.
I20260927 15:20:16.378935 10518 tablet_bootstrap.cc:654] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:16.380330 10518 log.cc:824] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:16.383544 10514 raft_consensus.cc:348] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.383437 10515 raft_consensus.cc:348] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.384250 10514 raft_consensus.cc:374] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:16.384397 10514 raft_consensus.cc:729] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33c24343375466f8861a09c8e6dc5d1, State: Initialized, Role: FOLLOWER
I20260927 15:20:16.384680 10515 raft_consensus.cc:374] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:16.384909 10515 raft_consensus.cc:729] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c607a92dcead4686badf99847336fcd8, State: Initialized, Role: FOLLOWER
I20260927 15:20:16.385633 10514 consensus_queue.cc:260] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.385970 10515 consensus_queue.cc:260] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.390259 10514 ts_tablet_manager.cc:1428] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260927 15:20:16.390679 10336 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
I20260927 15:20:16.397081 10518 tablet_bootstrap.cc:492] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: No bootstrap required, opened a new log
I20260927 15:20:16.397442 10518 ts_tablet_manager.cc:1397] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.005s
I20260927 15:20:16.407863 10515 ts_tablet_manager.cc:1428] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Time spent starting tablet: real 0.042s user 0.025s sys 0.010s
I20260927 15:20:16.408833 10478 heartbeater.cc:503] Master 127.2.246.126:35349 was elected leader, sending a full tablet report...
I20260927 15:20:16.409613 10518 raft_consensus.cc:348] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.410197 10518 raft_consensus.cc:374] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:16.410292 10518 raft_consensus.cc:729] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11e7745dc864f68b0cad8fefdf23540, State: Initialized, Role: FOLLOWER
I20260927 15:20:16.411466 10518 consensus_queue.cc:260] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
W20260927 15:20:16.414338 10338 tablet.cc:2369] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:16.419943 10518 ts_tablet_manager.cc:1428] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
W20260927 15:20:16.477699 10189 tablet.cc:2369] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:20:16.573952 10480 tablet.cc:2369] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:16.604185 10522 raft_consensus.cc:482] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:16.604482 10522 raft_consensus.cc:504] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.606336 10522 leader_election.cc:288] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785), b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:16.614866 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.607588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:51785 (local address 127.2.246.66:43861)
0927 15:20:16.607903 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.607910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.608218 (+ 308us) server_negotiation.cc:407] Connection header received
0927 15:20:16.608400 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.608405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.608492 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.608617 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:16.609634 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.610623 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.612835 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.613647 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.613888 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.613965 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.614435 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.614566 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 15:20:16.614856 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.606846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:41119 (local address 127.2.246.67:35785)
0927 15:20:16.607128 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.607134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.608045 (+ 911us) server_negotiation.cc:407] Connection header received
0927 15:20:16.608360 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.608367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.608472 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.608618 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:16.609272 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.610305 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.612763 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.613627 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.613819 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.613963 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.614356 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.614517 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260927 15:20:16.615006 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.606585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35785 (local address 127.2.246.65:41119)
0927 15:20:16.607871 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.607904 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.608106 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.608661 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.608666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.608691 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:16.609095 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.609106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.610469 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.610475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.612558 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.612577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.613914 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.613921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.614127 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.614200 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.614288 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.614450 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":6,"thread_start_us":879,"threads_started":1}
I20260927 15:20:16.615172 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.606844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43861 (local address 127.2.246.65:51785)
0927 15:20:16.608122 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.608137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:16.608242 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:16.608873 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:16.608878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:16.608907 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:16.609386 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.609400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.610831 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.610836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.612614 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:16.612629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.613975 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.613985 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:16.614217 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.614245 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:20:16.614376 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:20:16.614527 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":90,"thread_start_us":987,"threads_started":1}
I20260927 15:20:16.616216 10432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c21c45337c64389bc02782200ec2a2f" candidate_uuid: "c11e7745dc864f68b0cad8fefdf23540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c607a92dcead4686badf99847336fcd8" is_pre_election: true
I20260927 15:20:16.616438 10290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c21c45337c64389bc02782200ec2a2f" candidate_uuid: "c11e7745dc864f68b0cad8fefdf23540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33c24343375466f8861a09c8e6dc5d1" is_pre_election: true
I20260927 15:20:16.616864 10432 raft_consensus.cc:2454] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c11e7745dc864f68b0cad8fefdf23540 in term 0.
I20260927 15:20:16.617062 10290 raft_consensus.cc:2454] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c11e7745dc864f68b0cad8fefdf23540 in term 0.
I20260927 15:20:16.617620 10077 leader_election.cc:302] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c11e7745dc864f68b0cad8fefdf23540, c607a92dcead4686badf99847336fcd8; no voters:
I20260927 15:20:16.618106 10522 raft_consensus.cc:2781] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:16.618261 10522 raft_consensus.cc:482] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:16.618381 10522 raft_consensus.cc:3037] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:16.621034 10522 raft_consensus.cc:504] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.621743 10522 leader_election.cc:288] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [CANDIDATE]: Term 1 election: Requested vote from peers c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785), b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861)
I20260927 15:20:16.622524 10432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c21c45337c64389bc02782200ec2a2f" candidate_uuid: "c11e7745dc864f68b0cad8fefdf23540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c607a92dcead4686badf99847336fcd8"
I20260927 15:20:16.622778 10432 raft_consensus.cc:3037] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:16.622761 10290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c21c45337c64389bc02782200ec2a2f" candidate_uuid: "c11e7745dc864f68b0cad8fefdf23540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33c24343375466f8861a09c8e6dc5d1"
I20260927 15:20:16.623116 10290 raft_consensus.cc:3037] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:16.625291 10432 raft_consensus.cc:2454] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c11e7745dc864f68b0cad8fefdf23540 in term 1.
I20260927 15:20:16.625941 10290 raft_consensus.cc:2454] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c11e7745dc864f68b0cad8fefdf23540 in term 1.
I20260927 15:20:16.625954 10077 leader_election.cc:302] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c11e7745dc864f68b0cad8fefdf23540, c607a92dcead4686badf99847336fcd8; no voters:
I20260927 15:20:16.626341 10522 raft_consensus.cc:2781] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:16.626886 10522 raft_consensus.cc:686] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 1 LEADER]: Becoming Leader. State: Replica: c11e7745dc864f68b0cad8fefdf23540, State: Running, Role: LEADER
I20260927 15:20:16.627424 10522 consensus_queue.cc:237] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } }
I20260927 15:20:16.632468 9217 catalog_manager.cc:5495] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 reported cstate change: term changed from 0 to 1, leader changed from <none> to c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "c11e7745dc864f68b0cad8fefdf23540" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } health_report { overall_health: UNKNOWN } } }
I20260927 15:20:16.683390 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.676607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.2.246.65:46669)
0927 15:20:16.676862 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.676871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.677123 (+ 252us) server_negotiation.cc:407] Connection header received
0927 15:20:16.677426 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.677431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.677531 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.677717 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:16.678285 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.679060 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.680770 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.681231 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.681367 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.682063 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:16.682710 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:16.682847 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.683079 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.683154 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 15:20:16.694198 10290 raft_consensus.cc:1261] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Refusing update from remote peer c11e7745dc864f68b0cad8fefdf23540: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:16.694288 10432 raft_consensus.cc:1261] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Refusing update from remote peer c11e7745dc864f68b0cad8fefdf23540: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:16.695580 10525 consensus_queue.cc:1035] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:16.696056 10522 consensus_queue.cc:1035] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:16.712329 10529 mvcc.cc:205] Tried to move back new op lower bound from 7333979818766897152 to 7333979818509746176. Current Snapshot: MvccSnapshot[applied={T|T < 7333979818766897152}]
I20260927 15:20:16.715659 10530 mvcc.cc:205] Tried to move back new op lower bound from 7333979818766897152 to 7333979818509746176. Current Snapshot: MvccSnapshot[applied={T|T < 7333979818766897152}]
I20260927 15:20:16.749404 9216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:16.749806 9216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:16.759188 9216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01abce2181f74267bccc54e24211dd8d: Sending DeleteTablet for 3 replicas of tablet 5c21c45337c64389bc02782200ec2a2f
I20260927 15:20:16.760550 10412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c21c45337c64389bc02782200ec2a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:16 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:44834
I20260927 15:20:16.760740 10270 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c21c45337c64389bc02782200ec2a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:16 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:49776
I20260927 15:20:16.760712 10121 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c21c45337c64389bc02782200ec2a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:16 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:33866
I20260927 15:20:16.776768 10539 tablet_replica.cc:330] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: stopping tablet replica
I20260927 15:20:16.777622 10539 raft_consensus.cc:2229] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:16.778781 10540 tablet_replica.cc:330] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: stopping tablet replica
I20260927 15:20:16.779271 10540 raft_consensus.cc:2229] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:16.780051 10539 raft_consensus.cc:2258] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:16.781024 10540 raft_consensus.cc:2258] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:16.782140 10538 tablet_replica.cc:330] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: stopping tablet replica
I20260927 15:20:16.782761 10538 raft_consensus.cc:2229] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:16.783406 10538 raft_consensus.cc:2258] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:16.784232 10540 ts_tablet_manager.cc:1905] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:16.786314 10538 ts_tablet_manager.cc:1905] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:16.790675 10539 ts_tablet_manager.cc:1905] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:16.795006 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9185
I20260927 15:20:16.796486 9247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:16.803895 10540 ts_tablet_manager.cc:1918] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:16.804177 10540 log.cc:1196] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/5c21c45337c64389bc02782200ec2a2f
I20260927 15:20:16.804186 10538 ts_tablet_manager.cc:1918] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:16.804380 10538 log.cc:1196] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/5c21c45337c64389bc02782200ec2a2f
I20260927 15:20:16.804867 10540 ts_tablet_manager.cc:1939] T 5c21c45337c64389bc02782200ec2a2f P c11e7745dc864f68b0cad8fefdf23540: Deleting consensus metadata
I20260927 15:20:16.804937 10538 ts_tablet_manager.cc:1939] T 5c21c45337c64389bc02782200ec2a2f P c607a92dcead4686badf99847336fcd8: Deleting consensus metadata
I20260927 15:20:16.810628 10539 ts_tablet_manager.cc:1918] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:16.810930 10539 log.cc:1196] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/5c21c45337c64389bc02782200ec2a2f
I20260927 15:20:16.811659 10539 ts_tablet_manager.cc:1939] T 5c21c45337c64389bc02782200ec2a2f P b33c24343375466f8861a09c8e6dc5d1: Deleting consensus metadata
I20260927 15:20:16.821357 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.766094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.2.246.125:37729)
0927 15:20:16.792943 (+ 26849us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.792949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.797300 (+ 4351us) server_negotiation.cc:407] Connection header received
0927 15:20:16.797383 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.797476 (+ 93us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.797596 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.800117 (+ 2521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.800296 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.801370 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.803323 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.804020 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.808066 (+ 4046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.808344 (+ 278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.808351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.808356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.808396 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.813612 (+ 5216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.818918 (+ 5306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.818931 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.818933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.819150 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.819631 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.819635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.819637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.820018 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.820336 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.820646 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.820937 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26675,"thread_start_us":26550,"threads_started":1}
I20260927 15:20:16.837935 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.794094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.2.246.124:41739)
0927 15:20:16.813157 (+ 19063us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.813166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:16.821743 (+ 8577us) server_negotiation.cc:407] Connection header received
0927 15:20:16.821859 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:16.821867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:16.821984 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:16.823755 (+ 1771us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:16.823935 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.824725 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.826097 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:16.826547 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:16.826791 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:16.830616 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:16.830624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:16.830628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:16.830677 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:16.833761 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.834619 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.834625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.834629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.834776 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:16.836098 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:16.836105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:16.836108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:16.836556 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:16.836943 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:16.837219 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 15:20:16.837485 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18875,"thread_start_us":19088,"threads_started":1}
I20260927 15:20:16.965631 9203 catalog_manager.cc:4841] TS b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861): tablet 5c21c45337c64389bc02782200ec2a2f (table test-table [id=3b33e498600c4bbba859d8f979b6790a]) successfully deleted
I20260927 15:20:16.967114 9204 catalog_manager.cc:4841] TS c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669): tablet 5c21c45337c64389bc02782200ec2a2f (table test-table [id=3b33e498600c4bbba859d8f979b6790a]) successfully deleted
I20260927 15:20:16.967530 9204 catalog_manager.cc:4841] TS c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785): tablet 5c21c45337c64389bc02782200ec2a2f (table test-table [id=3b33e498600c4bbba859d8f979b6790a]) successfully deleted
I20260927 15:20:16.967914 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9185
I20260927 15:20:17.022179 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.999653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.2.246.124:41739)
0927 15:20:16.999923 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:16.999932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:17.001636 (+ 1704us) server_negotiation.cc:407] Connection header received
0927 15:20:17.002051 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:17.002061 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:17.002185 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:17.002312 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:17.006879 (+ 4567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:17.007996 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:17.009763 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:17.010341 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:17.010504 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:17.014351 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:17.014356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:17.014359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:17.014407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:17.018541 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:17.019456 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:17.019461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:17.019464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:17.019669 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:17.020120 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:17.020125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:17.020128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:17.020547 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:17.020841 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:17.021430 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 15:20:17.021744 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:20:17.026832 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.000690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.2.246.125:37729)
0927 15:20:17.000911 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:17.000919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:17.007430 (+ 6511us) server_negotiation.cc:407] Connection header received
0927 15:20:17.007973 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:17.007979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:17.008078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:17.008197 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:17.011869 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:17.013172 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:17.015629 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:17.016326 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:17.016488 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:17.019343 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:17.019355 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:17.019362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:17.019421 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:17.023268 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:17.024693 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:17.024697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:17.024700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:17.024839 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:17.025347 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:17.025352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:17.025354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:17.025698 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:17.026028 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:17.026365 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:20:17.026559 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:20:17.728665 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.727714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:59597)
0927 15:20:17.728303 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.728431 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":342,"threads_started":1}
W20260927 15:20:17.729152 10187 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:35349 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:17.737872 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.736952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:49765)
0927 15:20:17.737507 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.737646 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
W20260927 15:20:17.738276 10336 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:35349 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:17.745815 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.744782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:45313)
0927 15:20:17.745354 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.745548 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
W20260927 15:20:17.746423 10478 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:35349 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.258836 10555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01abce2181f74267bccc54e24211dd8d)
I20260927 15:20:18.259091 10556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01abce2181f74267bccc54e24211dd8d)
I20260927 15:20:18.259346 10556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:18.259191 10555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:18.260746 10556 leader_election.cc:288] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125:37729)
I20260927 15:20:18.261318 10555 leader_election.cc:288] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124:41739)
I20260927 15:20:18.263000 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.261260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57722)
0927 15:20:18.262363 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.262649 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":657,"threads_started":1}
I20260927 15:20:18.268889 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.261509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57734)
0927 15:20:18.268054 (+ 6545us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.268405 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6355,"threads_started":1}
W20260927 15:20:18.273746 9995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.274345 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.262508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:37729 (local address 127.0.0.1:39794)
0927 15:20:18.263208 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.263235 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:18.263425 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:18.264551 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:18.264560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:18.264607 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:18.264980 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:18.265028 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.267752 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.267757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:18.270637 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:18.270663 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.273176 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.273186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:18.273486 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:18.273550 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 15:20:18.273715 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:20:18.273938 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":10760,"threads_started":1}
I20260927 15:20:18.274451 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.262918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.2.246.125:37729)
0927 15:20:18.264137 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:18.264147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:18.264172 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:20:18.264262 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:18.264270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:18.264366 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:18.265172 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:18.265428 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.267023 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.270959 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.272942 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.273111 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:18.273253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:18.273776 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 15:20:18.274016 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":808,"threads_started":1}
W20260927 15:20:18.278676 9804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.281406 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.261516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:41739 (local address 127.0.0.1:56276)
0927 15:20:18.269189 (+ 7673us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.269217 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:18.269583 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:18.270578 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:18.270586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:18.270639 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:18.271162 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:18.271181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.274730 (+ 3549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.274742 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:18.277123 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:18.277144 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.280138 (+ 2994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.280150 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:18.280481 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:18.280546 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 15:20:18.280687 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 15:20:18.280932 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7549,"mutex_wait_us":140,"thread_start_us":5971,"threads_started":1}
I20260927 15:20:18.283046 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.261977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.2.246.124:41739)
0927 15:20:18.267646 (+ 5669us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:18.267652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:18.269442 (+ 1790us) server_negotiation.cc:407] Connection header received
0927 15:20:18.269863 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:18.269884 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:18.270010 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:18.270125 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:18.271481 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.272942 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.277409 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:18.279628 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:18.279836 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:18.281867 (+ 2031us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:18.281984 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 15:20:18.282232 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5479,"thread_start_us":5329,"threads_started":1}
I20260927 15:20:18.283571 9826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "276a84cd677549cbacf13a50293fdfb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1503936f32ae4a87ac1c183c9a1b0308" is_pre_election: true
I20260927 15:20:18.283897 10018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1503936f32ae4a87ac1c183c9a1b0308" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "276a84cd677549cbacf13a50293fdfb6" is_pre_election: true
I20260927 15:20:18.284083 9826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276a84cd677549cbacf13a50293fdfb6 in term 1.
I20260927 15:20:18.284276 10018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1503936f32ae4a87ac1c183c9a1b0308 in term 1.
I20260927 15:20:18.284907 9803 leader_election.cc:302] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1503936f32ae4a87ac1c183c9a1b0308, 276a84cd677549cbacf13a50293fdfb6; no voters:
I20260927 15:20:18.285108 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.284468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57746)
0927 15:20:18.284684 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.284811 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:20:18.285370 10555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:20:18.285517 10555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01abce2181f74267bccc54e24211dd8d)
I20260927 15:20:18.285655 10555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 1 FOLLOWER]: Advancing to term 2
W20260927 15:20:18.287254 9995 leader_election.cc:334] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.287715 9995 leader_election.cc:302] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1503936f32ae4a87ac1c183c9a1b0308, 276a84cd677549cbacf13a50293fdfb6; no voters: 01abce2181f74267bccc54e24211dd8d
I20260927 15:20:18.288158 10556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:20:18.288313 10556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01abce2181f74267bccc54e24211dd8d)
I20260927 15:20:18.288430 10556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:20:18.289615 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.289145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57766)
0927 15:20:18.289311 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.289395 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:20:18.289875 9804 leader_election.cc:334] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.291164 10555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:18.292189 10555 leader_election.cc:288] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 election: Requested vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 276a84cd677549cbacf13a50293fdfb6 (127.2.246.124:41739)
I20260927 15:20:18.292076 10556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:18.292723 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.292225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57776)
0927 15:20:18.292456 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.292559 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 15:20:18.292928 10556 leader_election.cc:288] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 election: Requested vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125:37729)
I20260927 15:20:18.292964 10018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1503936f32ae4a87ac1c183c9a1b0308" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "276a84cd677549cbacf13a50293fdfb6"
I20260927 15:20:18.293309 10018 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1503936f32ae4a87ac1c183c9a1b0308 in current term 2: Already voted for candidate 276a84cd677549cbacf13a50293fdfb6 in this term.
I20260927 15:20:18.293723 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.293135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57784)
0927 15:20:18.293398 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.293514 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 15:20:18.293794 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.293361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57788)
0927 15:20:18.293553 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.293643 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 15:20:18.293932 9826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "276a84cd677549cbacf13a50293fdfb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1503936f32ae4a87ac1c183c9a1b0308"
W20260927 15:20:18.294024 9804 leader_election.cc:334] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.294310 9826 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 276a84cd677549cbacf13a50293fdfb6 in current term 2: Already voted for candidate 1503936f32ae4a87ac1c183c9a1b0308 in this term.
I20260927 15:20:18.294667 9803 leader_election.cc:302] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1503936f32ae4a87ac1c183c9a1b0308; no voters: 01abce2181f74267bccc54e24211dd8d, 276a84cd677549cbacf13a50293fdfb6
I20260927 15:20:18.295073 10555 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:20:18.295218 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.294664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:57800)
0927 15:20:18.294919 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.295035 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 15:20:18.295450 9995 leader_election.cc:334] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:18.295643 9995 leader_election.cc:302] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 276a84cd677549cbacf13a50293fdfb6; no voters: 01abce2181f74267bccc54e24211dd8d, 1503936f32ae4a87ac1c183c9a1b0308
I20260927 15:20:18.295939 10556 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:20:18.731149 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.730181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:34599)
0927 15:20:18.730846 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.730964 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":344,"threads_started":1}
I20260927 15:20:18.740321 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.739190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:57001)
0927 15:20:18.739885 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.740032 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":385,"threads_started":1}
I20260927 15:20:18.748189 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.747290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:36411)
0927 15:20:18.747875 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.747992 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
I20260927 15:20:19.733390 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.732391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:44113)
0927 15:20:19.733048 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.733171 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":335,"threads_started":1}
I20260927 15:20:19.742568 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.741653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:42179)
0927 15:20:19.742203 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.742314 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":337,"threads_started":1}
I20260927 15:20:19.750332 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.749404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:39621)
0927 15:20:19.750033 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.750151 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":298,"threads_started":1}
I20260927 15:20:20.079166 10571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:20.079432 10571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:20.080199 10571 leader_election.cc:288] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125:37729)
I20260927 15:20:20.081401 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.080516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58048)
0927 15:20:20.081074 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.081210 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260927 15:20:20.081882 9826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "276a84cd677549cbacf13a50293fdfb6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1503936f32ae4a87ac1c183c9a1b0308" is_pre_election: true
I20260927 15:20:20.082212 9826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276a84cd677549cbacf13a50293fdfb6 in term 2.
I20260927 15:20:20.083181 9995 leader_election.cc:302] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1503936f32ae4a87ac1c183c9a1b0308, 276a84cd677549cbacf13a50293fdfb6; no voters:
I20260927 15:20:20.083498 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.082876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58056)
0927 15:20:20.083068 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.083169 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 15:20:20.083730 9995 leader_election.cc:334] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:20.083753 10571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 15:20:20.083959 10571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:20.084120 10571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:20:20.086835 10571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:20.087659 10571 leader_election.cc:288] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 election: Requested vote from peers 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349), 1503936f32ae4a87ac1c183c9a1b0308 (127.2.246.125:37729)
I20260927 15:20:20.088447 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.087968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58070)
0927 15:20:20.088147 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.088275 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:20:20.088864 9826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "276a84cd677549cbacf13a50293fdfb6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1503936f32ae4a87ac1c183c9a1b0308"
I20260927 15:20:20.089102 9826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:20:20.090091 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.089362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58080)
0927 15:20:20.089623 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.089721 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:20:20.090310 9995 leader_election.cc:334] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
I20260927 15:20:20.091780 9826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276a84cd677549cbacf13a50293fdfb6 in term 3.
I20260927 15:20:20.092514 9995 leader_election.cc:302] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1503936f32ae4a87ac1c183c9a1b0308, 276a84cd677549cbacf13a50293fdfb6; no voters: 01abce2181f74267bccc54e24211dd8d
I20260927 15:20:20.092939 10571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:20:20.093657 10571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [term 3 LEADER]: Becoming Leader. State: Replica: 276a84cd677549cbacf13a50293fdfb6, State: Running, Role: LEADER
I20260927 15:20:20.094400 10571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } }
I20260927 15:20:20.097218 10574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276a84cd677549cbacf13a50293fdfb6. Latest consensus state: current_term: 3 leader_uuid: "276a84cd677549cbacf13a50293fdfb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:20.097671 10574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:20.098668 10576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:20.103091 10576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:20.104415 10576 catalog_manager.cc:1259] Loaded cluster ID: bfa39b6ae0a64801b3a56667a890e0de
I20260927 15:20:20.104625 10576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:20.106245 10576 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:20.107371 10576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Loaded TSK: 0
I20260927 15:20:20.108494 10576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:20.121645 10007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:20.131247 10007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:20.136965 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.136340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58098)
0927 15:20:20.136577 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.136697 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:20:20.137380 9826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [term 3 FOLLOWER]: Refusing update from remote peer 276a84cd677549cbacf13a50293fdfb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 15:20:20.138521 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.138060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58104)
0927 15:20:20.138237 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.138328 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:20:20.138540 10574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 15:20:20.138931 9995 consensus_peers.cc:487] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 -> Peer 01abce2181f74267bccc54e24211dd8d (127.2.246.126:35349): Couldn't send request to peer 01abce2181f74267bccc54e24211dd8d. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:20:20.142316 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276a84cd677549cbacf13a50293fdfb6. Latest consensus state: current_term: 3 leader_uuid: "276a84cd677549cbacf13a50293fdfb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:20.142769 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:20.144532 10571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "276a84cd677549cbacf13a50293fdfb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:20.144897 10571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:20.146092 10579 mvcc.cc:205] Tried to move back new op lower bound from 7333979832872099840 to 7333979832712241152. Current Snapshot: MvccSnapshot[applied={T|T < 7333979832872099840}]
I20260927 15:20:20.147260 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "276a84cd677549cbacf13a50293fdfb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "01abce2181f74267bccc54e24211dd8d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35349 } } peers { permanent_uuid: "1503936f32ae4a87ac1c183c9a1b0308" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 37729 } attrs { promote: false } } peers { permanent_uuid: "276a84cd677549cbacf13a50293fdfb6" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 41739 } attrs { promote: false } } }
I20260927 15:20:20.147689 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1503936f32ae4a87ac1c183c9a1b0308 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:20:20.178138 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.158638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.2.246.66:43861)
0927 15:20:20.159340 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.159350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.168698 (+ 9348us) server_negotiation.cc:407] Connection header received
0927 15:20:20.170096 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.170106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.170227 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.170320 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:20.171597 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.173064 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.175654 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.176920 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.177073 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.177187 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.177597 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.177726 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":829,"threads_started":1}
I20260927 15:20:20.178194 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.158294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43861 (local address 127.0.0.1:49778)
0927 15:20:20.168506 (+ 10212us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.168543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:20.169884 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:20.170677 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:20.170685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:20.170719 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:20.171320 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.171338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.173296 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.173302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.175437 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.175452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.177282 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.177286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.177422 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.177445 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:20:20.177562 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:20:20.177722 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9878,"thread_start_us":9722,"threads_started":1}
I20260927 15:20:20.179260 10270 tablet_service.cc:1467] Processing CreateTablet for tablet b85794355fa84540a85eef6fa4a4599b (DEFAULT_TABLE table=test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:20.179785 10270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85794355fa84540a85eef6fa4a4599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:20.181041 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.157784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35785 (local address 127.0.0.1:44842)
0927 15:20:20.158100 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.158134 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:20.158297 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:20.173738 (+ 15441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:20.173747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:20.173805 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:20.174365 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.174376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.175991 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.175999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.178302 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.178321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.180164 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.180172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.180387 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.180423 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:20:20.180509 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:20:20.180673 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":22}
I20260927 15:20:20.181041 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.157875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.2.246.67:35785)
0927 15:20:20.173296 (+ 15421us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.173302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.173322 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:20.173396 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.173401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.173500 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.173619 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:20.174596 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.175785 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.178564 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.179903 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.180340 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.180467 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.180545 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.180705 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15249,"thread_start_us":15862,"threads_started":1}
I20260927 15:20:20.182888 10412 tablet_service.cc:1467] Processing CreateTablet for tablet b85794355fa84540a85eef6fa4a4599b (DEFAULT_TABLE table=test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:20.183566 10412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85794355fa84540a85eef6fa4a4599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:20.184610 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.159146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46669 (local address 127.0.0.1:33896)
0927 15:20:20.169216 (+ 10070us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.169239 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:20.169420 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:20.174740 (+ 5320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:20.174745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:20.174773 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:20.176252 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.176270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.177958 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.177963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.181463 (+ 3500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.181475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.183572 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.183580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.183797 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.183834 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:20:20.183940 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:20:20.184112 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9915,"mutex_wait_us":4,"thread_start_us":10404,"threads_started":1}
I20260927 15:20:20.185441 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.159796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.2.246.65:46669)
0927 15:20:20.174381 (+ 14585us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.174387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.174408 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:20:20.174488 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.174493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.174576 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.174693 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:20.176544 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.177745 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.181671 (+ 3926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.183048 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.183209 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.183391 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.184889 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.185082 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14414,"thread_start_us":14239,"threads_started":1}
I20260927 15:20:20.186260 10589 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Bootstrap starting.
I20260927 15:20:20.187903 10589 tablet_bootstrap.cc:654] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:20.190279 10591 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Bootstrap starting.
I20260927 15:20:20.190873 10121 tablet_service.cc:1467] Processing CreateTablet for tablet b85794355fa84540a85eef6fa4a4599b (DEFAULT_TABLE table=test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:20.191345 10589 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: No bootstrap required, opened a new log
I20260927 15:20:20.191525 10121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85794355fa84540a85eef6fa4a4599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:20.191524 10589 ts_tablet_manager.cc:1397] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 15:20:20.192382 10591 tablet_bootstrap.cc:654] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:20.192302 10589 raft_consensus.cc:348] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.192674 10589 raft_consensus.cc:374] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:20.192778 10589 raft_consensus.cc:729] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33c24343375466f8861a09c8e6dc5d1, State: Initialized, Role: FOLLOWER
I20260927 15:20:20.193060 10589 consensus_queue.cc:260] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.195351 10591 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: No bootstrap required, opened a new log
I20260927 15:20:20.195545 10591 ts_tablet_manager.cc:1397] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:20.196631 10591 raft_consensus.cc:348] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.196983 10591 raft_consensus.cc:374] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:20.197094 10591 raft_consensus.cc:729] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c607a92dcead4686badf99847336fcd8, State: Initialized, Role: FOLLOWER
I20260927 15:20:20.197384 10591 consensus_queue.cc:260] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.198936 10595 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Bootstrap starting.
I20260927 15:20:20.200846 10595 tablet_bootstrap.cc:654] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:20.200999 10589 ts_tablet_manager.cc:1428] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20260927 15:20:20.203495 10595 tablet_bootstrap.cc:492] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: No bootstrap required, opened a new log
I20260927 15:20:20.203598 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.203121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:47083)
0927 15:20:20.203293 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.203415 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:20:20.203758 10595 ts_tablet_manager.cc:1397] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:20.204747 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.204293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:40641)
0927 15:20:20.204481 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.204573 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 15:20:20.204841 10595 raft_consensus.cc:348] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.205130 10595 raft_consensus.cc:374] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:20.205236 10595 raft_consensus.cc:729] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11e7745dc864f68b0cad8fefdf23540, State: Initialized, Role: FOLLOWER
I20260927 15:20:20.205546 10595 consensus_queue.cc:260] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.207015 10591 ts_tablet_manager.cc:1428] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Time spent starting tablet: real 0.011s user 0.002s sys 0.007s
I20260927 15:20:20.207629 10335 heartbeater.cc:503] Master 127.2.246.124:41739 was elected leader, sending a full tablet report...
I20260927 15:20:20.207983 10595 ts_tablet_manager.cc:1428] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:20.209463 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.208785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:32995)
0927 15:20:20.209183 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.209291 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260927 15:20:20.210685 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.210097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:45213)
0927 15:20:20.210425 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.210539 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260927 15:20:20.211064 10477 heartbeater.cc:503] Master 127.2.246.124:41739 was elected leader, sending a full tablet report...
I20260927 15:20:20.211216 10186 heartbeater.cc:503] Master 127.2.246.124:41739 was elected leader, sending a full tablet report...
I20260927 15:20:20.212976 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.212230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:40759)
0927 15:20:20.212633 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.212750 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 15:20:20.453120 10593 raft_consensus.cc:482] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:20.453351 10593 raft_consensus.cc:504] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.454913 10593 leader_election.cc:288] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785), c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
I20260927 15:20:20.463299 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.455422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35785 (local address 127.2.246.66:57335)
0927 15:20:20.455660 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.455686 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:20.455861 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:20.456490 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:20.456497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:20.456530 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:20.456888 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.456901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.458271 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.458275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.460453 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.460467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.462299 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.462308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.462543 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.462613 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 15:20:20.462738 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:20:20.462898 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":60}
I20260927 15:20:20.463330 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.455536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:57335 (local address 127.2.246.67:35785)
0927 15:20:20.455797 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.455804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.455826 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:20.456154 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.456163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.456284 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.456409 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:20.457062 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.458140 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.460653 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.462012 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.462233 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.462347 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.462861 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.463014 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:20:20.463330 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.456172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36721 (local address 127.2.246.65:46669)
0927 15:20:20.456452 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.456459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.456801 (+ 342us) server_negotiation.cc:407] Connection header received
0927 15:20:20.457005 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.457012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.457106 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.457218 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:20.457670 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.458600 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.460897 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.462207 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.462360 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.462437 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.462861 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.463014 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:20:20.463373 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.455879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46669 (local address 127.2.246.66:36721)
0927 15:20:20.456668 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.456685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:20.456822 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:20.457235 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:20.457238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:20.457256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:20.457522 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.457529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.458723 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.458728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.460739 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:20.460748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.462442 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.462448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:20.462655 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.462695 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:20:20.462780 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:20:20.462897 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":110,"thread_start_us":523,"threads_started":1}
I20260927 15:20:20.464790 10432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85794355fa84540a85eef6fa4a4599b" candidate_uuid: "b33c24343375466f8861a09c8e6dc5d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c607a92dcead4686badf99847336fcd8" is_pre_election: true
I20260927 15:20:20.464849 10141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85794355fa84540a85eef6fa4a4599b" candidate_uuid: "b33c24343375466f8861a09c8e6dc5d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11e7745dc864f68b0cad8fefdf23540" is_pre_election: true
I20260927 15:20:20.465085 10432 raft_consensus.cc:2454] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b33c24343375466f8861a09c8e6dc5d1 in term 0.
I20260927 15:20:20.465318 10141 raft_consensus.cc:2454] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b33c24343375466f8861a09c8e6dc5d1 in term 0.
I20260927 15:20:20.465895 10226 leader_election.cc:302] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b33c24343375466f8861a09c8e6dc5d1, c607a92dcead4686badf99847336fcd8; no voters:
I20260927 15:20:20.466410 10593 raft_consensus.cc:2781] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:20.466560 10593 raft_consensus.cc:482] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:20.466688 10593 raft_consensus.cc:3037] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:20.468986 10593 raft_consensus.cc:504] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.469614 10593 leader_election.cc:288] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [CANDIDATE]: Term 1 election: Requested vote from peers c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785), c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669)
I20260927 15:20:20.470351 10432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85794355fa84540a85eef6fa4a4599b" candidate_uuid: "b33c24343375466f8861a09c8e6dc5d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c607a92dcead4686badf99847336fcd8"
I20260927 15:20:20.470604 10432 raft_consensus.cc:3037] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:20.471061 10141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85794355fa84540a85eef6fa4a4599b" candidate_uuid: "b33c24343375466f8861a09c8e6dc5d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11e7745dc864f68b0cad8fefdf23540"
I20260927 15:20:20.471311 10141 raft_consensus.cc:3037] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:20.472931 10432 raft_consensus.cc:2454] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b33c24343375466f8861a09c8e6dc5d1 in term 1.
I20260927 15:20:20.473551 10226 leader_election.cc:302] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b33c24343375466f8861a09c8e6dc5d1, c607a92dcead4686badf99847336fcd8; no voters:
I20260927 15:20:20.474200 10593 raft_consensus.cc:2781] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:20.474395 10141 raft_consensus.cc:2454] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b33c24343375466f8861a09c8e6dc5d1 in term 1.
I20260927 15:20:20.474820 10593 raft_consensus.cc:686] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 1 LEADER]: Becoming Leader. State: Replica: b33c24343375466f8861a09c8e6dc5d1, State: Running, Role: LEADER
I20260927 15:20:20.475874 10593 consensus_queue.cc:237] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } }
I20260927 15:20:20.479256 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.478734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:57155)
0927 15:20:20.478927 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.479048 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 15:20:20.480335 10007 catalog_manager.cc:5495] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "b33c24343375466f8861a09c8e6dc5d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b33c24343375466f8861a09c8e6dc5d1" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 } health_report { overall_health: UNKNOWN } } }
I20260927 15:20:20.480607 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.480057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:60287)
0927 15:20:20.480287 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.480413 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:20:20.562939 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.556974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.2.246.66:43861)
0927 15:20:20.557199 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.557204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.557224 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:20.557341 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.557345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.557475 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.557685 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:20.558104 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.558889 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.560271 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.560753 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.560875 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.561373 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:20.562196 (+ 823us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:20.562332 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.562549 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.562656 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:20:20.568220 10432 raft_consensus.cc:1261] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Refusing update from remote peer b33c24343375466f8861a09c8e6dc5d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:20.568588 10141 raft_consensus.cc:1261] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Refusing update from remote peer b33c24343375466f8861a09c8e6dc5d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:20.569159 10599 consensus_queue.cc:1035] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c607a92dcead4686badf99847336fcd8" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:20.569685 10593 consensus_queue.cc:1035] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11e7745dc864f68b0cad8fefdf23540" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:20.572803 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.572311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:37533)
0927 15:20:20.572472 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.572583 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 15:20:20.573769 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.573109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:56921)
0927 15:20:20.573374 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.573506 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 15:20:20.579377 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.578948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.67:47247)
0927 15:20:20.579102 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.579195 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:20:20.579836 10603 mvcc.cc:205] Tried to move back new op lower bound from 7333979834638925824 to 7333979834274050048. Current Snapshot: MvccSnapshot[applied={T|T < 7333979834638925824}]
I20260927 15:20:20.581420 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.580845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:41569)
0927 15:20:20.581076 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.581216 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 15:20:20.594439 10007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:20.594789 10007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:20.599115 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.598507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.66:45599)
0927 15:20:20.598775 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.598907 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 15:20:20.601027 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.600432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.2.246.65:57407)
0927 15:20:20.600617 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.600748 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:20:20.605247 10007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 276a84cd677549cbacf13a50293fdfb6: Sending DeleteTablet for 3 replicas of tablet b85794355fa84540a85eef6fa4a4599b
I20260927 15:20:20.606809 10412 tablet_service.cc:1514] Processing DeleteTablet for tablet b85794355fa84540a85eef6fa4a4599b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:20 UTC) from {username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:44842
I20260927 15:20:20.606815 10270 tablet_service.cc:1514] Processing DeleteTablet for tablet b85794355fa84540a85eef6fa4a4599b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:20 UTC) from {username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:49778
I20260927 15:20:20.609654 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9980
I20260927 15:20:20.613416 10611 tablet_replica.cc:330] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: stopping tablet replica
I20260927 15:20:20.613716 10611 raft_consensus.cc:2229] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:20.615347 10611 raft_consensus.cc:2258] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:20.617072 10121 tablet_service.cc:1514] Processing DeleteTablet for tablet b85794355fa84540a85eef6fa4a4599b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:20 UTC) from {username='oryx', principal='oryx/127.2.246.124@KRBTEST.COM'} at 127.0.0.1:33896
I20260927 15:20:20.617748 10611 ts_tablet_manager.cc:1905] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:20.626397 10614 tablet_replica.cc:330] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: stopping tablet replica
I20260927 15:20:20.626853 10614 raft_consensus.cc:2229] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:20.627394 10614 raft_consensus.cc:2258] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:20.629403 10614 ts_tablet_manager.cc:1905] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:20.631836 10612 tablet_replica.cc:330] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: stopping tablet replica
I20260927 15:20:20.632107 10612 raft_consensus.cc:2229] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:20.632488 10612 raft_consensus.cc:2258] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:20.634438 10611 ts_tablet_manager.cc:1918] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:20.634620 10611 log.cc:1196] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/b85794355fa84540a85eef6fa4a4599b
I20260927 15:20:20.635355 10611 ts_tablet_manager.cc:1939] T b85794355fa84540a85eef6fa4a4599b P b33c24343375466f8861a09c8e6dc5d1: Deleting consensus metadata
I20260927 15:20:20.636386 10612 ts_tablet_manager.cc:1905] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:20.636976 10614 ts_tablet_manager.cc:1918] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:20.637122 10614 log.cc:1196] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/b85794355fa84540a85eef6fa4a4599b
I20260927 15:20:20.637671 10614 ts_tablet_manager.cc:1939] T b85794355fa84540a85eef6fa4a4599b P c11e7745dc864f68b0cad8fefdf23540: Deleting consensus metadata
I20260927 15:20:20.637595 9994 catalog_manager.cc:4841] TS b33c24343375466f8861a09c8e6dc5d1 (127.2.246.66:43861): tablet b85794355fa84540a85eef6fa4a4599b (table test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]) successfully deleted
I20260927 15:20:20.638898 9995 catalog_manager.cc:4841] TS c11e7745dc864f68b0cad8fefdf23540 (127.2.246.65:46669): tablet b85794355fa84540a85eef6fa4a4599b (table test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]) successfully deleted
I20260927 15:20:20.645773 10612 ts_tablet_manager.cc:1918] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:20.646003 10612 log.cc:1196] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/b85794355fa84540a85eef6fa4a4599b
I20260927 15:20:20.646516 10612 ts_tablet_manager.cc:1939] T b85794355fa84540a85eef6fa4a4599b P c607a92dcead4686badf99847336fcd8: Deleting consensus metadata
I20260927 15:20:20.648031 9995 catalog_manager.cc:4841] TS c607a92dcead4686badf99847336fcd8 (127.2.246.67:35785): tablet b85794355fa84540a85eef6fa4a4599b (table test-table [id=82ebfed4157f4aab9119f0b22cfb4d87]) successfully deleted
I20260927 15:20:20.650945 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.614083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.2.246.65:46669)
0927 15:20:20.614379 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.614389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.631665 (+ 17276us) server_negotiation.cc:407] Connection header received
0927 15:20:20.633501 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.633510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.633736 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.633874 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:20.637756 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.638896 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.640150 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.640542 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.640695 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.643331 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:20.643358 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:20.643375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:20.643414 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:20.646726 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:20.647463 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:20.647472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:20.647474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:20.647605 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:20.648205 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:20.648214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:20.648221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:20.648894 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:20.649287 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.649782 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.650493 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 15:20:20.651794 10181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:20.739936 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.738843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58106)
0927 15:20:20.739476 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.739645 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260927 15:20:20.741775 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.741312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58122)
0927 15:20:20.741500 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.741609 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 15:20:20.816622 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9980
I20260927 15:20:20.840662 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 10198
I20260927 15:20:20.860276 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.841598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.2.246.66:43861)
0927 15:20:20.841783 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:20.841788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:20.841813 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:20:20.842130 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:20.842134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:20.842246 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:20.842364 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:20.844518 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.845390 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.847602 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:20.848161 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:20.848312 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:20.852828 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:20.852847 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:20.852889 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:20.852974 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:20.856541 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:20.857763 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:20.857771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:20.857774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:20.858050 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:20.858579 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:20.858583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:20.858585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:20.858983 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:20.859249 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:20.859738 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 15:20:20.859937 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:20:20.860965 10330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:21.008795 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 10198
I20260927 15:20:21.031615 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 10348
I20260927 15:20:21.053561 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.032611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.2.246.67:35785)
0927 15:20:21.033108 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:21.033112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:21.033131 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:20:21.033194 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:21.033198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:21.033268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:21.033390 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:21.035905 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:21.036709 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:21.038307 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:21.038778 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:21.038900 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:21.046288 (+ 7388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:21.046308 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:21.046325 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:21.046384 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:21.050307 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:21.051239 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:21.051247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:21.051250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:21.051477 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:21.051992 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:21.051996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:21.051998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:21.052366 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:21.052641 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:21.053040 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 15:20:21.053239 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260927 15:20:21.054325 10472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:21.139957 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.139344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58136)
0927 15:20:21.139590 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:21.139738 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:20:21.141356 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.140895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35349 (local address 127.0.0.1:58146)
0927 15:20:21.141084 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:21.141204 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:20:21.166038 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 10348
I20260927 15:20:21.185494 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9680
I20260927 15:20:21.187060 9847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:21.308817 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9680
I20260927 15:20:21.330811 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 9782
I20260927 15:20:21.332080 10038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:21.454937 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 9782
2026-09-27T15:20:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (13311 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:21 dist-test-slave-14jq krb5kdc[10625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:21 dist-test-slave-14jq krb5kdc[10625](info): set up 2 sockets
Sep 27 15:20:21 dist-test-slave-14jq krb5kdc[10625](info): commencing operation
krb5kdc: starting...
W20260927 15:20:22.599414 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:20:22 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab.
Entry for principal oryx/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab.
I20260927 15:20:22.866654 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46389
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab
--principal=oryx/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:23.296702 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:23.297173 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:23.297361 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:23.307199 10638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:23.307415 10638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:20:23.307523 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:23.307642 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:23.307747 10638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:23.307862 10638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:23.320350 10638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:46389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.126.keytab
--principal=oryx/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:23.322322 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:23.324098 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:23.340948 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:23.340971 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:23.342196 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:23.342603 10638 server_base.cc:1034] running on GCE node
I20260927 15:20:23.344323 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:23.346128 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:23.347482 10638 hybrid_clock.cc:648] HybridClock initialized: now 1790522423347375 us; error 80 us; skew 500 ppm
I20260927 15:20:23.347910 10638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:23 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522423, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:23.351805 10638 init.cc:377] Logged in from keytab as oryx/127.2.246.126@KRBTEST.COM (short username oryx)
I20260927 15:20:23.354071 10638 webserver.cc:460] Webserver started at http://127.2.246.126:37321/ using document root <none> and password file <none>
I20260927 15:20:23.354995 10638 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:23.355144 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:23.355518 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:23.359045 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "5dfeba54145e4d05ba06cf6877760543"
format_stamp: "Formatted at 2026-09-27 15:20:23 on dist-test-slave-14jq"
server_key: "4d4beb838ced78a517443c88236ea870"
server_key_iv: "172c5cd7988cab55db921fe2659c01b4"
server_key_version: "encryptionkey@0"
I20260927 15:20:23.359985 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "5dfeba54145e4d05ba06cf6877760543"
format_stamp: "Formatted at 2026-09-27 15:20:23 on dist-test-slave-14jq"
server_key: "4d4beb838ced78a517443c88236ea870"
server_key_iv: "172c5cd7988cab55db921fe2659c01b4"
server_key_version: "encryptionkey@0"
I20260927 15:20:23.367000 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 15:20:23.372090 10653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:23.374161 10638 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:23.374473 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "5dfeba54145e4d05ba06cf6877760543"
format_stamp: "Formatted at 2026-09-27 15:20:23 on dist-test-slave-14jq"
server_key: "4d4beb838ced78a517443c88236ea870"
server_key_iv: "172c5cd7988cab55db921fe2659c01b4"
server_key_version: "encryptionkey@0"
I20260927 15:20:23.374756 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:23.393672 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:23.398646 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:23.398937 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:23.425779 10638 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:46389
I20260927 15:20:23.425933 10714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:46389 every 8 connection(s)
I20260927 15:20:23.427690 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:23.433274 10715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:23.435596 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 10638
I20260927 15:20:23.435863 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:23.436686 3033 external_mini_cluster.cc:1428] Setting key 6761c1a9a6c7528f3d6e16a20944825a
I20260927 15:20:23.449641 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Bootstrap starting.
Sep 27 15:20:23 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:23.456666 10715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:23.458365 10715 log.cc:824] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:23.462558 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: No bootstrap required, opened a new log
I20260927 15:20:23.468178 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:23.439296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.2.246.126:46389)
0927 15:20:23.440521 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:23.440536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:23.440599 (+ 63us) server_negotiation.cc:407] Connection header received
0927 15:20:23.442684 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:23.442713 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:23.443223 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:23.443682 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:23.445977 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:23.447535 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:23.449555 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:23.450556 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:23.450756 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:23.457697 (+ 6941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:23.457713 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:23.457740 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:23.457866 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:23.462099 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:23.463301 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:23.463310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:23.463329 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:23.463614 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:23.464159 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:23.464163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:23.464166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:23.464984 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:23.465292 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:23.465804 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:20:23.466525 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":519,"threads_started":1}
I20260927 15:20:23.472056 10715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } }
I20260927 15:20:23.472536 10715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:23.472673 10715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfeba54145e4d05ba06cf6877760543, State: Initialized, Role: FOLLOWER
I20260927 15:20:23.473349 10715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } }
I20260927 15:20:23.473588 10715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:23.473757 10715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:23.473999 10715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:23.476334 10715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } }
I20260927 15:20:23.476977 10715 leader_election.cc:302] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dfeba54145e4d05ba06cf6877760543; no voters:
I20260927 15:20:23.477733 10715 leader_election.cc:288] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:23.478019 10720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:23.479323 10720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [term 1 LEADER]: Becoming Leader. State: Replica: 5dfeba54145e4d05ba06cf6877760543, State: Running, Role: LEADER
I20260927 15:20:23.480422 10715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:23.480408 10720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } }
I20260927 15:20:23.484819 10721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dfeba54145e4d05ba06cf6877760543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } } }
I20260927 15:20:23.485392 10721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:23.484862 10722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dfeba54145e4d05ba06cf6877760543. Latest consensus state: current_term: 1 leader_uuid: "5dfeba54145e4d05ba06cf6877760543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dfeba54145e4d05ba06cf6877760543" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46389 } } }
I20260927 15:20:23.485579 10722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543 [sys.catalog]: This master's current role is: LEADER
W20260927 15:20:23.500865 10736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:23.501020 10736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:23.501197 10726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:23.503780 10726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:23.515717 10726 catalog_manager.cc:1347] Generated new cluster ID: c1e608f707384c24bd8edbf26064c405
I20260927 15:20:23.515867 10726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:23.529422 10726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:23.531020 10726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:23.541641 10726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Generated new TSK 0
I20260927 15:20:23.542681 10726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
I20260927 15:20:23.610785 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46389
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:24.031531 10741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:24.031957 10741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:24.032182 10741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:24.041738 10741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:24.042078 10741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:24.042268 10741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:20:24.042430 10741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:24.055423 10741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:46389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:24.057461 10741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:24.059545 10741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:24.076988 10747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:24.077006 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:24.076941 10746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:24.078156 10741 server_base.cc:1034] running on GCE node
I20260927 15:20:24.079124 10741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:24.080538 10741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:24.081930 10741 hybrid_clock.cc:648] HybridClock initialized: now 1790522424081801 us; error 46 us; skew 500 ppm
I20260927 15:20:24.082394 10741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:24 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522424, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:24.086872 10741 init.cc:377] Logged in from keytab as oryx/127.2.246.65@KRBTEST.COM (short username oryx)
I20260927 15:20:24.089124 10741 webserver.cc:460] Webserver started at http://127.2.246.65:33763/ using document root <none> and password file <none>
I20260927 15:20:24.090533 10741 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:24.090711 10741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:24.091231 10741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:24.095162 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "f029dc78567f45308977558185937334"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "132d7a46f83fb20362c2602f55547ef6"
server_key_iv: "9db8260ae16852dbaa54d2d0d93a4c95"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.096230 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "f029dc78567f45308977558185937334"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "132d7a46f83fb20362c2602f55547ef6"
server_key_iv: "9db8260ae16852dbaa54d2d0d93a4c95"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.103435 10741 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:20:24.108067 10756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:24.110087 10741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:24.110368 10741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "f029dc78567f45308977558185937334"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "132d7a46f83fb20362c2602f55547ef6"
server_key_iv: "9db8260ae16852dbaa54d2d0d93a4c95"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.110657 10741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:24.124457 10741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:24.130544 10741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:24.130911 10741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:24.134539 10741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:24.134675 10741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:24.134866 10741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:24.134953 10741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:20:24 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522424, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:24.159588 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.135182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:57491 (local address 127.2.246.126:46389)
0927 15:20:24.138442 (+ 3260us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:24.138447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:24.138468 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:20:24.141606 (+ 3138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:24.141615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:24.141753 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:24.141921 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:24.144964 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.145996 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.147864 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.148278 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.148422 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.152765 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:24.152771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:24.152774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:24.152807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:24.156122 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.156913 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.156922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.156927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.157088 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.157590 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.157597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.157603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.157960 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:24.158104 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:24.158971 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 15:20:24.159209 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3134,"thread_start_us":3037,"threads_started":1}
I20260927 15:20:24.160799 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.136102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.65:57491)
0927 15:20:24.137340 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.137412 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:24.140868 (+ 3456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:24.142434 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:24.142455 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:24.143571 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:24.144714 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.144743 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.146210 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.146214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.147701 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.147715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.148541 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.148548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.148772 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.149507 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:24.149530 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:24.152427 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:24.156300 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.156310 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.156329 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.156783 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.157236 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.157247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.157249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.157465 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.158104 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:24.158120 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:24.158594 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 15:20:24.158872 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 15:20:24.159451 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":511,"threads_started":1}
I20260927 15:20:24.192564 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.173489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:41445 (local address 127.2.246.126:46389)
0927 15:20:24.173786 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:24.173795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:24.174579 (+ 784us) server_negotiation.cc:407] Connection header received
0927 15:20:24.174917 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:24.174927 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:24.175075 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:24.175288 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:24.176271 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.177908 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.181046 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.181997 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.182222 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.184313 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:24.184323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:24.184332 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:24.184389 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:24.189294 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.190380 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.190388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.190392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.190590 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.190989 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.190999 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.191001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.191325 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:24.191497 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:24.191940 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 15:20:24.192202 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:20:24.192675 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.173274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.65:41445)
0927 15:20:24.174392 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.174413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:24.174636 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:24.175327 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:24.175331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:24.175763 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:24.176074 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.176083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.178177 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.178186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.180648 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.180673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.182302 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.182311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.182505 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.183213 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:24.183232 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:24.184021 (+ 789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:24.189572 (+ 5551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.189577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.189581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.190189 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.190725 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.190730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.190734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.190861 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.191456 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:24.191462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:24.191653 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 15:20:24.191816 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:20:24.192202 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":731,"threads_started":1}
I20260927 15:20:24.213487 10741 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:35053
I20260927 15:20:24.213553 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:35053 every 8 connection(s)
I20260927 15:20:24.215140 10741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:24.217926 10877 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46389
I20260927 15:20:24.218235 10877 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:24.218990 10877 heartbeater.cc:511] Master 127.2.246.126:46389 requested a full tablet report, sending...
I20260927 15:20:24.219980 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 10741
I20260927 15:20:24.220238 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:24.220996 3033 external_mini_cluster.cc:1428] Setting key 3907506cd215982948e84a057f7e54dc
I20260927 15:20:24.221324 10670 ts_manager.cc:194] Registered new tserver with Master: f029dc78567f45308977558185937334 (127.2.246.65:35053)
I20260927 15:20:24.223331 10670 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:57491
WARNING: no policy specified for oryx/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
I20260927 15:20:24.278493 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46389
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:24.648828 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:24.649156 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:24.649297 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:24.656898 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:24.657030 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:24.657176 10881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:20:24.657337 10881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:24.667932 10881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:46389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:24.669426 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:24.670971 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:24.687026 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:24.687151 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:24.687420 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:24.689254 10881 server_base.cc:1034] running on GCE node
I20260927 15:20:24.690166 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:24.691262 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:24.692749 10881 hybrid_clock.cc:648] HybridClock initialized: now 1790522424692599 us; error 112 us; skew 500 ppm
I20260927 15:20:24.693445 10881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:24 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522424, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:24.697497 10881 init.cc:377] Logged in from keytab as oryx/127.2.246.66@KRBTEST.COM (short username oryx)
I20260927 15:20:24.699462 10881 webserver.cc:460] Webserver started at http://127.2.246.66:41135/ using document root <none> and password file <none>
I20260927 15:20:24.700361 10881 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:24.700512 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:24.700863 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:24.704073 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "17d26ff89c1441d0991310262aa4e4e3"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "5345c28c6caeb142540facf202e13e9c"
server_key_iv: "c9203b3d0e31c131d21981859f6af343"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.704912 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "17d26ff89c1441d0991310262aa4e4e3"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "5345c28c6caeb142540facf202e13e9c"
server_key_iv: "c9203b3d0e31c131d21981859f6af343"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.711642 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:20:24.716073 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:24.718156 10881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260927 15:20:24.718384 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "17d26ff89c1441d0991310262aa4e4e3"
format_stamp: "Formatted at 2026-09-27 15:20:24 on dist-test-slave-14jq"
server_key: "5345c28c6caeb142540facf202e13e9c"
server_key_iv: "c9203b3d0e31c131d21981859f6af343"
server_key_version: "encryptionkey@0"
I20260927 15:20:24.718642 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:24.736971 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:24.741788 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:24.742244 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:24.745594 10881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:24.745710 10881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:24.745918 10881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:24.746003 10881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 15:20:24 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522424, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:24.780114 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.746826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:46067 (local address 127.2.246.126:46389)
0927 15:20:24.747456 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:24.747467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:24.749320 (+ 1853us) server_negotiation.cc:407] Connection header received
0927 15:20:24.753410 (+ 4090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:24.753417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:24.753526 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:24.753673 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:24.756976 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.757986 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.761126 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.761943 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.762148 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.769716 (+ 7568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:24.769725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:24.769733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:24.769786 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:24.774518 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.776055 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.776062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.776065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.776290 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.777153 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.777158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.777160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.777525 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:24.777707 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:24.779370 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0927 15:20:24.779729 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
I20260927 15:20:24.781950 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.747835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.66:46067)
0927 15:20:24.749037 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.749122 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:24.752953 (+ 3831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:24.754061 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:24.754080 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:24.755316 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:24.756633 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.756675 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.758238 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.758247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.760859 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.760887 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.762401 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.762414 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.762892 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.763927 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:24.763978 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:24.769237 (+ 5259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:24.774792 (+ 5555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.774811 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.774834 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.775760 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.776508 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.776517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.776524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.776929 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.777727 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:24.777741 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:24.778546 (+ 805us) client_negotiation.cc:769] Sending connection context
0927 15:20:24.778973 (+ 427us) client_negotiation.cc:240] Negotiation successful
0927 15:20:24.779803 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":510,"threads_started":1}
I20260927 15:20:24.810187 10881 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:42799
I20260927 15:20:24.810266 11016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:42799 every 8 connection(s)
I20260927 15:20:24.812144 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:24.814838 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.793684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36883 (local address 127.2.246.126:46389)
0927 15:20:24.794215 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:24.794223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:24.794516 (+ 293us) server_negotiation.cc:407] Connection header received
0927 15:20:24.794788 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:24.794799 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:24.794923 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:24.795091 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:24.797347 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.799127 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.801710 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.802696 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.802944 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.805911 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:24.805922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:24.805931 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:24.805997 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:24.810084 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.811142 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.811149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.811152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.811339 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:24.811844 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:24.811849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:24.811852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:24.812383 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:24.812538 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:24.813179 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 15:20:24.813544 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260927 15:20:24.814159 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.793114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.66:36883)
0927 15:20:24.794317 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.794353 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:24.794587 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:24.795176 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:24.795184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:24.796236 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:24.797066 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.797086 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.799474 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.799479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.801422 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:24.801447 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:24.803006 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:24.803016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:24.803272 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:24.804101 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:24.804132 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:24.805513 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:24.810339 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.810348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.810353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.810971 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.811509 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:24.811513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:24.811515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:24.811721 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:24.812529 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:24.812533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:24.812780 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 15:20:24.813015 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 15:20:24.813478 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":809,"threads_started":1}
I20260927 15:20:24.815155 11017 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46389
I20260927 15:20:24.815409 11017 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:24.816663 11017 heartbeater.cc:511] Master 127.2.246.126:46389 requested a full tablet report, sending...
I20260927 15:20:24.818356 10670 ts_manager.cc:194] Registered new tserver with Master: 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:24.819278 10670 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:46067
I20260927 15:20:24.820272 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 10881
I20260927 15:20:24.820451 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:24.820990 3033 external_mini_cluster.cc:1428] Setting key 796fe8a646849b687e2586d828cb14b6
WARNING: no policy specified for oryx/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
I20260927 15:20:24.878327 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46389
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:25.227195 10877 heartbeater.cc:503] Master 127.2.246.126:46389 was elected leader, sending a full tablet report...
W20260927 15:20:25.304445 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:25.304841 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:25.305040 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:25.313510 11021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:25.313669 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:25.313858 11021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:20:25.313983 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:25.325448 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.246.126:46389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:25.327252 11021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:25.329010 11021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:25.346662 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:25.346869 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:25.347380 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:25.348523 11021 server_base.cc:1034] running on GCE node
I20260927 15:20:25.349427 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:25.350769 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:25.352061 11021 hybrid_clock.cc:648] HybridClock initialized: now 1790522425351978 us; error 39 us; skew 500 ppm
I20260927 15:20:25.352522 11021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:25 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522425, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:25.356532 11021 init.cc:377] Logged in from keytab as oryx/127.2.246.67@KRBTEST.COM (short username oryx)
I20260927 15:20:25.358770 11021 webserver.cc:460] Webserver started at http://127.2.246.67:46429/ using document root <none> and password file <none>
I20260927 15:20:25.359753 11021 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:25.359920 11021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:25.360344 11021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:25.364326 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "8e40a8fbb6d64d7cb1d7a829c873604d"
format_stamp: "Formatted at 2026-09-27 15:20:25 on dist-test-slave-14jq"
server_key: "ff86988fa986a7e9dc1a434dc009e1ec"
server_key_iv: "933b2450f7989624dee42dde4034100f"
server_key_version: "encryptionkey@0"
I20260927 15:20:25.365453 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "8e40a8fbb6d64d7cb1d7a829c873604d"
format_stamp: "Formatted at 2026-09-27 15:20:25 on dist-test-slave-14jq"
server_key: "ff86988fa986a7e9dc1a434dc009e1ec"
server_key_iv: "933b2450f7989624dee42dde4034100f"
server_key_version: "encryptionkey@0"
I20260927 15:20:25.372666 11021 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:20:25.378104 11036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:25.380074 11021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:20:25.380257 11021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "8e40a8fbb6d64d7cb1d7a829c873604d"
format_stamp: "Formatted at 2026-09-27 15:20:25 on dist-test-slave-14jq"
server_key: "ff86988fa986a7e9dc1a434dc009e1ec"
server_key_iv: "933b2450f7989624dee42dde4034100f"
server_key_version: "encryptionkey@0"
I20260927 15:20:25.380486 11021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:25.394224 11021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:25.398874 11021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:25.399125 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:25.402354 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:25.402452 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:25.402617 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:25.402688 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:20:25 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522425, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:25.441707 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.403981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:50571 (local address 127.2.246.126:46389)
0927 15:20:25.404525 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:25.404531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:25.406438 (+ 1907us) server_negotiation.cc:407] Connection header received
0927 15:20:25.410811 (+ 4373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:25.410822 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:25.410957 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:25.411138 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:25.415011 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.416407 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.420436 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.421280 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.421481 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:25.430917 (+ 9436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:25.430928 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:25.430934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:25.431003 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:25.435654 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:25.437260 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:25.437267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:25.437272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:25.437484 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:25.438531 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:25.438538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:25.438541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:25.439140 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:25.439383 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:25.440617 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0927 15:20:25.441023 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260927 15:20:25.442646 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.404715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.67:50571)
0927 15:20:25.406189 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:25.406268 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:25.410240 (+ 3972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:25.411620 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:25.411639 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:25.412931 (+ 1292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:25.414462 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:25.414515 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.416759 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.416768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:25.420087 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:25.420127 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.422000 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.422016 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:25.422567 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:25.423907 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:25.423962 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:25.430321 (+ 6359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:25.435973 (+ 5652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:25.435992 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:25.436028 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:25.437004 (+ 976us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:25.437633 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:25.437638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:25.437642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:25.438239 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:25.439367 (+ 1128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:25.439378 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:25.440055 (+ 677us) client_negotiation.cc:769] Sending connection context
0927 15:20:25.440383 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 15:20:25.440975 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":600,"threads_started":1}
I20260927 15:20:25.455849 11021 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:45251
I20260927 15:20:25.455880 11156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:45251 every 8 connection(s)
I20260927 15:20:25.457645 11021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:25.459964 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11021
I20260927 15:20:25.460181 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:25.460790 3033 external_mini_cluster.cc:1428] Setting key d5acb2a583ac8dc3f6306967ea23cbc6
I20260927 15:20:25.460610 11157 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46389
I20260927 15:20:25.461910 11157 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:25.462911 11157 heartbeater.cc:511] Master 127.2.246.126:46389 requested a full tablet report, sending...
I20260927 15:20:25.465094 10669 ts_manager.cc:194] Registered new tserver with Master: 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251)
I20260927 15:20:25.466643 10669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:50571
I20260927 15:20:25.468921 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.452242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:46427 (local address 127.2.246.126:46389)
0927 15:20:25.452532 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:25.452542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:25.452955 (+ 413us) server_negotiation.cc:407] Connection header received
0927 15:20:25.453226 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:25.453237 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:25.453349 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:25.453487 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:25.454753 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.456029 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.457543 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.458296 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.458488 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:25.461203 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:25.461209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:25.461213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:25.461251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:25.465341 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:25.466536 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:25.466541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:25.466543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:25.466689 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:25.467159 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:25.467163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:25.467165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:25.467632 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:25.467852 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:25.468440 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 15:20:25.468656 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:20:25.469350 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.452103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.2.246.67:46427)
0927 15:20:25.452789 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:25.452819 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:25.453035 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:25.453537 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:25.453540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:25.454073 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:25.454528 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:25.454544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.456248 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.456256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:25.457365 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:25.457378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.458601 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.458612 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:25.458839 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:25.459762 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:25.459791 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:25.460983 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:25.465536 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:25.465542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:25.465548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:25.466325 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:25.466830 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:25.466835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:25.466839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:25.467024 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:25.467819 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:25.467825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:25.468070 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 15:20:25.468267 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:20:25.468689 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":370,"threads_started":1}
I20260927 15:20:25.479056 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:20:25 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:20:25 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:25 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:25.822721 11017 heartbeater.cc:503] Master 127.2.246.126:46389 was elected leader, sending a full tablet report...
I20260927 15:20:26.471091 11157 heartbeater.cc:503] Master 127.2.246.126:46389 was elected leader, sending a full tablet report...
W20260927 15:20:26.646138 11165 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.106s user 0.010s sys 0.009s
W20260927 15:20:26.647550 11045 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:25.533677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.2.246.126:46389)
0927 15:20:25.533949 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:25.533954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:25.534339 (+ 385us) server_negotiation.cc:407] Connection header received
0927 15:20:25.534519 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:25.534524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:25.534631 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:25.534760 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:25.536280 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.537081 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.538794 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:25.539328 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:25.539464 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.647212 (+1107748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42970: BlockingRecv error: recv got EOF from 127.0.0.1:42970 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 15:20:26 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:26.676591 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.653582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.2.246.126:46389)
0927 15:20:26.653890 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.653898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.654334 (+ 436us) server_negotiation.cc:407] Connection header received
0927 15:20:26.654511 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.654516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.654598 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.654754 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:26.656199 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.657032 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.658628 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.659188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.659333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.669712 (+ 10379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:26.669720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:26.669724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:26.669790 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:26.673306 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:26.674282 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:26.674287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:26.674290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:26.674464 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:26.674857 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:26.674861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:26.674863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:26.675194 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:26.675428 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.675896 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.676255 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 15:20:26.683287 10669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:26.692821 10669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:26.737751 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.709077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:35053 (local address 127.0.0.1:59434)
0927 15:20:26.722959 (+ 13882us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.722998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:26.724166 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:26.727287 (+ 3121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:26.727300 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:26.727354 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:26.727965 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.727979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.730148 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.730154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.733813 (+ 3659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.733865 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.736108 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.736114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.736271 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.736311 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:20:26.736765 (+ 454us) client_negotiation.cc:240] Negotiation successful
0927 15:20:26.737083 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13447,"mutex_wait_us":200,"thread_start_us":16867,"threads_started":1}
I20260927 15:20:26.738329 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.711008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.2.246.65:35053)
0927 15:20:26.725860 (+ 14852us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.725877 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.725939 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:20:26.726088 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.726102 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.726594 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.726904 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:26.728196 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.729945 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.734121 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.735577 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.735745 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.735959 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.737663 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.737967 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14528,"thread_start_us":14359,"threads_started":1}
I20260927 15:20:26.738677 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.709077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:42799 (local address 127.0.0.1:47814)
0927 15:20:26.726288 (+ 17211us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.726307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:26.726484 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:26.727702 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:26.727710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:26.727747 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:26.728277 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.728290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.730366 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.730373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.733864 (+ 3491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.733875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.737584 (+ 3709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.737590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.737878 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.737914 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:20:26.738016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:20:26.738164 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17062,"mutex_wait_us":610,"thread_start_us":12218,"threads_started":1}
I20260927 15:20:26.739445 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.710046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.2.246.67:45251)
0927 15:20:26.728245 (+ 18199us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.728257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.728289 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:20:26.728406 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.728414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.728627 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.728861 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:26.729432 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.731954 (+ 2522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.735057 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.737026 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.737174 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.737356 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.738928 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.739141 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17982,"thread_start_us":17819,"threads_started":1}
I20260927 15:20:26.739435 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.709366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.2.246.66:42799)
0927 15:20:26.725337 (+ 15971us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.725353 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.726618 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 15:20:26.726761 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.726774 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.727174 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.727459 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:26.728508 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.730165 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.734136 (+ 3971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.736973 (+ 2837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.737155 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.737356 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.738969 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.739146 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15664,"thread_start_us":15457,"threads_started":1}
I20260927 15:20:26.738785 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.709403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:45251 (local address 127.0.0.1:34520)
0927 15:20:26.722934 (+ 13531us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.722980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:26.723229 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:26.728847 (+ 5618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:26.728852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:26.728879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:26.729252 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.729261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.732117 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.732125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.734688 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.734712 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.737747 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.737753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.738197 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.738226 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:20:26.738310 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:20:26.738444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13200,"thread_start_us":12334,"threads_started":1}
I20260927 15:20:26.743547 10811 tablet_service.cc:1467] Processing CreateTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 (DEFAULT_TABLE table=test-table [id=a1fac7ed970748ebae2a7058c238b98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:26.743868 11087 tablet_service.cc:1467] Processing CreateTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 (DEFAULT_TABLE table=test-table [id=a1fac7ed970748ebae2a7058c238b98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:26.743870 10947 tablet_service.cc:1467] Processing CreateTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 (DEFAULT_TABLE table=test-table [id=a1fac7ed970748ebae2a7058c238b98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:26.745358 10811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576fcdb5cd154d4b93cc2961f89c7477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:26.745456 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576fcdb5cd154d4b93cc2961f89c7477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:26.745447 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576fcdb5cd154d4b93cc2961f89c7477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:26.763662 11182 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Bootstrap starting.
I20260927 15:20:26.766914 11184 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Bootstrap starting.
I20260927 15:20:26.768014 11183 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Bootstrap starting.
I20260927 15:20:26.768253 11182 tablet_bootstrap.cc:654] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:26.770370 11182 log.cc:824] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:26.771565 11184 tablet_bootstrap.cc:654] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:26.773085 11184 log.cc:824] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:26.773088 11183 tablet_bootstrap.cc:654] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:26.774809 11183 log.cc:824] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:26.775808 11182 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: No bootstrap required, opened a new log
I20260927 15:20:26.776072 11184 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: No bootstrap required, opened a new log
I20260927 15:20:26.776228 11182 ts_tablet_manager.cc:1397] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260927 15:20:26.776413 11184 ts_tablet_manager.cc:1397] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 15:20:26.778739 11183 tablet_bootstrap.cc:492] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: No bootstrap required, opened a new log
I20260927 15:20:26.779116 11183 ts_tablet_manager.cc:1397] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:20:26.785003 11184 raft_consensus.cc:348] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.785517 11184 raft_consensus.cc:374] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:26.785635 11184 raft_consensus.cc:729] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Initialized, Role: FOLLOWER
I20260927 15:20:26.786332 11184 consensus_queue.cc:260] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.789674 11182 raft_consensus.cc:348] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.790501 11182 raft_consensus.cc:374] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:26.790653 11182 raft_consensus.cc:729] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f029dc78567f45308977558185937334, State: Initialized, Role: FOLLOWER
I20260927 15:20:26.791445 11184 ts_tablet_manager.cc:1428] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Time spent starting tablet: real 0.015s user 0.002s sys 0.013s
I20260927 15:20:26.791601 11182 consensus_queue.cc:260] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.792629 11183 raft_consensus.cc:348] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.793331 11183 raft_consensus.cc:374] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:26.793521 11183 raft_consensus.cc:729] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Initialized, Role: FOLLOWER
I20260927 15:20:26.794507 11183 consensus_queue.cc:260] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.797300 11182 ts_tablet_manager.cc:1428] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260927 15:20:26.797785 11183 ts_tablet_manager.cc:1428] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
W20260927 15:20:26.816720 11018 tablet.cc:2369] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:26.885142 11190 raft_consensus.cc:482] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:26.885537 11190 raft_consensus.cc:504] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.887933 11190 leader_election.cc:288] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:26.894999 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.888487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:42781 (local address 127.2.246.65:35053)
0927 15:20:26.888710 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.888719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.889413 (+ 694us) server_negotiation.cc:407] Connection header received
0927 15:20:26.889672 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.889678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.889749 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.889868 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:26.890461 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.891160 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.893141 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.894047 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.894233 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.894309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.894587 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.894727 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:20:26.895166 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.888326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:35053 (local address 127.2.246.67:42781)
0927 15:20:26.889275 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.889297 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:26.889517 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:26.889901 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:26.889905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:26.889934 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:26.890301 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.890312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.891296 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.891303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.892973 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.892986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.894253 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.894259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.894423 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.894448 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:20:26.894563 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:20:26.894714 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":4,"thread_start_us":1083,"threads_started":1}
I20260927 15:20:26.895712 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.888992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:58381 (local address 127.2.246.66:42799)
0927 15:20:26.889203 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.889209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.889978 (+ 769us) server_negotiation.cc:407] Connection header received
0927 15:20:26.890261 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.890266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.890334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.890442 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:26.891021 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.891779 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.893668 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.894582 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.894804 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.894893 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.895275 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.895422 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:20:26.896317 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.888393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:42799 (local address 127.2.246.67:58381)
0927 15:20:26.889775 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.889794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:26.890084 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:26.890467 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:26.890472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:26.890498 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:26.890845 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.890853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.891932 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.891939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.893467 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:26.893480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.894834 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.894840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:26.895025 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.895050 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:20:26.895795 (+ 745us) client_negotiation.cc:240] Negotiation successful
0927 15:20:26.895969 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":482,"threads_started":1}
I20260927 15:20:26.896917 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576fcdb5cd154d4b93cc2961f89c7477" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334" is_pre_election: true
I20260927 15:20:26.897563 10832 raft_consensus.cc:2454] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:26.897529 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576fcdb5cd154d4b93cc2961f89c7477" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3" is_pre_election: true
I20260927 15:20:26.898252 10967 raft_consensus.cc:2454] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:26.898558 11040 leader_election.cc:302] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e40a8fbb6d64d7cb1d7a829c873604d, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:26.899096 11190 raft_consensus.cc:2781] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:26.899297 11190 raft_consensus.cc:482] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:26.899426 11190 raft_consensus.cc:3037] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:26.901999 11190 raft_consensus.cc:504] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.902994 11190 leader_election.cc:288] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Requested vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:26.903860 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576fcdb5cd154d4b93cc2961f89c7477" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334"
I20260927 15:20:26.904157 10832 raft_consensus.cc:3037] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:26.904127 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576fcdb5cd154d4b93cc2961f89c7477" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3"
I20260927 15:20:26.904381 10967 raft_consensus.cc:3037] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:26.906901 10967 raft_consensus.cc:2454] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:26.907200 10832 raft_consensus.cc:2454] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:26.907421 11039 leader_election.cc:302] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d; no voters:
I20260927 15:20:26.907953 11190 raft_consensus.cc:2781] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:26.908529 11190 raft_consensus.cc:686] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Becoming Leader. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Running, Role: LEADER
I20260927 15:20:26.909168 11190 consensus_queue.cc:237] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:26.915198 10669 catalog_manager.cc:5495] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67). New cstate: current_term: 1 leader_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } health_report { overall_health: HEALTHY } } }
I20260927 15:20:26.939989 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.934051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.2.246.67:45251)
0927 15:20:26.934307 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:26.934313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:26.934527 (+ 214us) server_negotiation.cc:407] Connection header received
0927 15:20:26.934817 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:26.934820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:26.934922 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:26.935091 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:26.935437 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.936150 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.937482 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:26.937930 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:26.938070 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:26.938638 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:26.939285 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:26.939500 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:26.939619 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 15:20:26.939703 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:20:26.950663 10832 raft_consensus.cc:1261] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:26.950965 10967 raft_consensus.cc:1261] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:26.951931 11193 consensus_queue.cc:1035] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:26.952416 11190 consensus_queue.cc:1035] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:20:26.962074 11158 tablet.cc:2369] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:26.969281 11198 mvcc.cc:205] Tried to move back new op lower bound from 7333979860776275968 to 7333979860624621568. Current Snapshot: MvccSnapshot[applied={T|T < 7333979860776275968}]
I20260927 15:20:26.992765 10670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:26.993065 10670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:26.998545 10670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Sending DeleteTablet for 3 replicas of tablet 576fcdb5cd154d4b93cc2961f89c7477
I20260927 15:20:26.999936 10811 tablet_service.cc:1514] Processing DeleteTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:26 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260927 15:20:27.000264 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:26 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 15:20:27.000490 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:26 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:47814
I20260927 15:20:27.000312 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet 576fcdb5cd154d4b93cc2961f89c7477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:26 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 15:20:27.003772 10670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:27.005985 10670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:27.009428 11208 tablet_replica.cc:330] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: stopping tablet replica
I20260927 15:20:27.010756 11208 raft_consensus.cc:2229] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:27.012071 11208 raft_consensus.cc:2258] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.018049 11207 tablet_replica.cc:330] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: stopping tablet replica
I20260927 15:20:27.018850 11207 raft_consensus.cc:2229] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:27.019383 10810 tablet_service.cc:1467] Processing CreateTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 (DEFAULT_TABLE table=test-table [id=94f47c19450445b7b099ff663c03ab84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.019775 11207 raft_consensus.cc:2258] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.020092 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85908b45e5a4d59a6eb52e70f6a4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.021802 11087 tablet_service.cc:1467] Processing CreateTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 (DEFAULT_TABLE table=test-table [id=94f47c19450445b7b099ff663c03ab84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.022507 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85908b45e5a4d59a6eb52e70f6a4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.023277 11207 ts_tablet_manager.cc:1905] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.023480 10947 tablet_service.cc:1467] Processing CreateTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 (DEFAULT_TABLE table=test-table [id=94f47c19450445b7b099ff663c03ab84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.024106 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85908b45e5a4d59a6eb52e70f6a4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.030529 11206 tablet_replica.cc:330] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: stopping tablet replica
I20260927 15:20:27.031105 11206 raft_consensus.cc:2229] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:27.031728 11206 raft_consensus.cc:2258] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.032003 11208 ts_tablet_manager.cc:1905] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.033303 11183 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Bootstrap starting.
I20260927 15:20:27.034217 11206 ts_tablet_manager.cc:1905] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.035368 11183 tablet_bootstrap.cc:654] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.040107 11208 ts_tablet_manager.cc:1918] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:27.040704 11182 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Bootstrap starting.
I20260927 15:20:27.042161 11206 ts_tablet_manager.cc:1918] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:27.042375 11206 log.cc:1196] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/576fcdb5cd154d4b93cc2961f89c7477
I20260927 15:20:27.043146 11182 tablet_bootstrap.cc:654] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.043278 11184 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Bootstrap starting.
I20260927 15:20:27.045214 11184 tablet_bootstrap.cc:654] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.046561 11183 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: No bootstrap required, opened a new log
I20260927 15:20:27.046739 11208 log.cc:1196] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/576fcdb5cd154d4b93cc2961f89c7477
I20260927 15:20:27.046882 11183 ts_tablet_manager.cc:1397] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260927 15:20:27.047446 11207 ts_tablet_manager.cc:1918] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:27.047652 11207 log.cc:1196] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/576fcdb5cd154d4b93cc2961f89c7477
I20260927 15:20:27.047814 11208 ts_tablet_manager.cc:1939] T 576fcdb5cd154d4b93cc2961f89c7477 P 17d26ff89c1441d0991310262aa4e4e3: Deleting consensus metadata
I20260927 15:20:27.048313 11207 ts_tablet_manager.cc:1939] T 576fcdb5cd154d4b93cc2961f89c7477 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting consensus metadata
I20260927 15:20:27.048560 11183 raft_consensus.cc:348] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.049196 11206 ts_tablet_manager.cc:1939] T 576fcdb5cd154d4b93cc2961f89c7477 P f029dc78567f45308977558185937334: Deleting consensus metadata
I20260927 15:20:27.050953 10654 catalog_manager.cc:4841] TS 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251): tablet 576fcdb5cd154d4b93cc2961f89c7477 (table test-table [id=a1fac7ed970748ebae2a7058c238b98c]) successfully deleted
W20260927 15:20:27.051762 10654 catalog_manager.cc:4808] TS 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251): delete failed for tablet 576fcdb5cd154d4b93cc2961f89c7477 because the tablet was not found. No further retry: Not found: Tablet not found: 576fcdb5cd154d4b93cc2961f89c7477
I20260927 15:20:27.052205 10656 catalog_manager.cc:4841] TS 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799): tablet 576fcdb5cd154d4b93cc2961f89c7477 (table test-table [id=a1fac7ed970748ebae2a7058c238b98c]) successfully deleted
I20260927 15:20:27.049541 11183 raft_consensus.cc:374] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.052784 11183 raft_consensus.cc:729] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.053193 11183 consensus_queue.cc:260] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.053664 10657 catalog_manager.cc:4841] TS f029dc78567f45308977558185937334 (127.2.246.65:35053): tablet 576fcdb5cd154d4b93cc2961f89c7477 (table test-table [id=a1fac7ed970748ebae2a7058c238b98c]) successfully deleted
I20260927 15:20:27.054006 11183 ts_tablet_manager.cc:1428] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 15:20:27.059197 11182 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: No bootstrap required, opened a new log
I20260927 15:20:27.059381 11182 ts_tablet_manager.cc:1397] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260927 15:20:27.059417 11184 tablet_bootstrap.cc:492] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: No bootstrap required, opened a new log
I20260927 15:20:27.059628 11184 ts_tablet_manager.cc:1397] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20260927 15:20:27.060375 11182 raft_consensus.cc:348] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.060696 11182 raft_consensus.cc:374] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.060775 11182 raft_consensus.cc:729] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f029dc78567f45308977558185937334, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.061084 11182 consensus_queue.cc:260] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.060962 11184 raft_consensus.cc:348] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.061288 11184 raft_consensus.cc:374] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.061384 11184 raft_consensus.cc:729] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.061697 11182 ts_tablet_manager.cc:1428] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:20:27.061685 11184 consensus_queue.cc:260] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.062578 11184 ts_tablet_manager.cc:1428] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:27.128293 11202 raft_consensus.cc:482] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:27.128619 11202 raft_consensus.cc:504] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.129596 11202 leader_election.cc:288] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:27.130259 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85908b45e5a4d59a6eb52e70f6a4395" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3" is_pre_election: true
I20260927 15:20:27.130452 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85908b45e5a4d59a6eb52e70f6a4395" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334" is_pre_election: true
I20260927 15:20:27.130563 10967 raft_consensus.cc:2454] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:27.130728 10832 raft_consensus.cc:2454] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:27.131157 11039 leader_election.cc:302] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d; no voters:
I20260927 15:20:27.131479 11202 raft_consensus.cc:2781] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:27.131661 11202 raft_consensus.cc:482] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:27.131776 11202 raft_consensus.cc:3037] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.134410 11202 raft_consensus.cc:504] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.135075 11202 leader_election.cc:288] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Requested vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:27.135715 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85908b45e5a4d59a6eb52e70f6a4395" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3"
I20260927 15:20:27.135774 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85908b45e5a4d59a6eb52e70f6a4395" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334"
I20260927 15:20:27.135973 10832 raft_consensus.cc:3037] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.135959 10967 raft_consensus.cc:3037] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.139231 10832 raft_consensus.cc:2454] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:27.139231 10967 raft_consensus.cc:2454] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:27.139853 11039 leader_election.cc:302] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:27.140331 11202 raft_consensus.cc:2781] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:27.140537 11202 raft_consensus.cc:686] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Becoming Leader. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Running, Role: LEADER
I20260927 15:20:27.141002 11202 consensus_queue.cc:237] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.144431 10669 catalog_manager.cc:5495] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67). New cstate: current_term: 1 leader_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } health_report { overall_health: HEALTHY } } }
Sep 27 15:20:27 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522423, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.126@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:27.219367 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.200355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46389 (local address 127.0.0.1:42998)
0927 15:20:27.201274 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.201343 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.201579 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.202316 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.202321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.203502 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:27.203909 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.203924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.205268 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.205274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.207444 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.207466 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.208490 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.208499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.208713 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.209724 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:27.209753 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:27.212989 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:27.216661 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:27.216678 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:27.216687 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:27.217136 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:27.217509 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:27.217516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:27.217519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:27.217658 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:27.218247 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:27.218256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:27.218430 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.218590 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.218918 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":407,"threads_started":1}
I20260927 15:20:27.219446 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.200547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.2.246.126:46389)
0927 15:20:27.201884 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.201892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.201938 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:20:27.201998 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.202002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.202070 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.202201 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:27.204196 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.205080 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.207727 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.208205 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.208348 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.213246 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:27.213253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:27.213257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:27.213295 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:27.216512 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:27.217255 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:27.217264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:27.217267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:27.217402 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:27.217761 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:27.217768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:27.217771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:27.218113 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:27.218262 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.218692 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.219058 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":1176,"thread_start_us":974,"threads_started":1}
W20260927 15:20:27.220999 10878 tablet.cc:2369] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:27.225160 10669 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:42998:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:20:27.226369 10669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:27.233662 10810 tablet_service.cc:1467] Processing CreateTablet for tablet e8563238674c437ea034a00bf7cbd09c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8343dae1bba848e3bb84d575631e5416]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:20:27.234277 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8563238674c437ea034a00bf7cbd09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.238754 10947 tablet_service.cc:1467] Processing CreateTablet for tablet e8563238674c437ea034a00bf7cbd09c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8343dae1bba848e3bb84d575631e5416]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:20:27.238754 11087 tablet_service.cc:1467] Processing CreateTablet for tablet e8563238674c437ea034a00bf7cbd09c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8343dae1bba848e3bb84d575631e5416]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:20:27.239364 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8563238674c437ea034a00bf7cbd09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.239357 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8563238674c437ea034a00bf7cbd09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.246827 11184 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: Bootstrap starting.
I20260927 15:20:27.248616 11184 tablet_bootstrap.cc:654] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.249423 11182 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: Bootstrap starting.
I20260927 15:20:27.249706 11183 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: Bootstrap starting.
I20260927 15:20:27.251927 11182 tablet_bootstrap.cc:654] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.252517 11183 tablet_bootstrap.cc:654] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.258168 11184 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: No bootstrap required, opened a new log
I20260927 15:20:27.258383 11184 ts_tablet_manager.cc:1397] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 15:20:27.259397 11184 raft_consensus.cc:348] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.259755 11184 raft_consensus.cc:374] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.259862 11184 raft_consensus.cc:729] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.260264 11184 consensus_queue.cc:260] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.261193 11184 ts_tablet_manager.cc:1428] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:27.261065 11188 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.261723 11188 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.264596 11182 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: No bootstrap required, opened a new log
I20260927 15:20:27.264873 11183 tablet_bootstrap.cc:492] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: No bootstrap required, opened a new log
I20260927 15:20:27.264921 11182 ts_tablet_manager.cc:1397] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260927 15:20:27.265133 11183 ts_tablet_manager.cc:1397] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260927 15:20:27.266034 11183 raft_consensus.cc:348] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.266353 11183 raft_consensus.cc:374] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.266158 11182 raft_consensus.cc:348] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.266484 11183 raft_consensus.cc:729] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.266645 11182 raft_consensus.cc:374] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.266796 11182 raft_consensus.cc:729] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f029dc78567f45308977558185937334, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.266793 11183 consensus_queue.cc:260] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.267261 11182 consensus_queue.cc:260] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.267625 11183 ts_tablet_manager.cc:1428] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:27.267850 11189 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.268242 11189 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.267452 11202 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.268576 11202 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.269410 11182 ts_tablet_manager.cc:1428] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:27.391116 11188 raft_consensus.cc:482] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:27.391394 11188 raft_consensus.cc:504] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.393298 11188 leader_election.cc:288] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251)
I20260927 15:20:27.402617 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.394438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:43679 (local address 127.2.246.65:35053)
0927 15:20:27.394639 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.394645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.394913 (+ 268us) server_negotiation.cc:407] Connection header received
0927 15:20:27.395353 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.395357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.395473 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.395574 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:27.396493 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.397651 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.400587 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.401475 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.401693 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.401786 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.402101 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.402261 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:20:27.402715 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.393629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:35053 (local address 127.2.246.66:43679)
0927 15:20:27.394733 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.394764 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.395103 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.395790 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.395797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.395824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:27.396287 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.396300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.397824 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.397830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.400314 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.400330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.401685 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.401691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.401924 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.401951 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.402070 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.402261 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":41,"thread_start_us":1347,"threads_started":1}
I20260927 15:20:27.403457 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.394006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:53843 (local address 127.2.246.67:45251)
0927 15:20:27.394252 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.394259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.395544 (+ 1285us) server_negotiation.cc:407] Connection header received
0927 15:20:27.395926 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.395936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.396030 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.396131 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:27.396859 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.397662 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.400567 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.401962 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.402166 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.402248 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.402953 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.403105 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:20:27.403729 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8563238674c437ea034a00bf7cbd09c" candidate_uuid: "17d26ff89c1441d0991310262aa4e4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334" is_pre_election: true
I20260927 15:20:27.404063 10832 raft_consensus.cc:2454] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d26ff89c1441d0991310262aa4e4e3 in term 0.
I20260927 15:20:27.404305 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.393641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:45251 (local address 127.2.246.66:53843)
0927 15:20:27.395393 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.395418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.395679 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.396306 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.396311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.396336 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:27.396638 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.396650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.397900 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.397907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.400313 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.400330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.402474 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.402483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.402729 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.402773 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.403712 (+ 939us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.403896 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"mutex_wait_us":280,"thread_start_us":754,"threads_started":1}
I20260927 15:20:27.404721 10900 leader_election.cc:302] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:27.405205 11188 raft_consensus.cc:2781] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:27.405334 11188 raft_consensus.cc:482] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:27.405436 11188 raft_consensus.cc:3037] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.405552 11107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8563238674c437ea034a00bf7cbd09c" candidate_uuid: "17d26ff89c1441d0991310262aa4e4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" is_pre_election: true
I20260927 15:20:27.406001 11107 raft_consensus.cc:2454] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d26ff89c1441d0991310262aa4e4e3 in term 0.
I20260927 15:20:27.408583 11188 raft_consensus.cc:504] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.409601 11188 leader_election.cc:288] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [CANDIDATE]: Term 1 election: Requested vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251)
I20260927 15:20:27.410256 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8563238674c437ea034a00bf7cbd09c" candidate_uuid: "17d26ff89c1441d0991310262aa4e4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334"
I20260927 15:20:27.410465 11107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8563238674c437ea034a00bf7cbd09c" candidate_uuid: "17d26ff89c1441d0991310262aa4e4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d"
I20260927 15:20:27.410609 10832 raft_consensus.cc:3037] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.410773 11107 raft_consensus.cc:3037] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.414170 10832 raft_consensus.cc:2454] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d26ff89c1441d0991310262aa4e4e3 in term 1.
I20260927 15:20:27.414343 11107 raft_consensus.cc:2454] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d26ff89c1441d0991310262aa4e4e3 in term 1.
I20260927 15:20:27.414961 10897 leader_election.cc:302] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d; no voters:
I20260927 15:20:27.415437 11188 raft_consensus.cc:2781] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:27.416002 11188 raft_consensus.cc:686] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Running, Role: LEADER
I20260927 15:20:27.416517 11188 consensus_queue.cc:237] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.419559 11225 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: TxnStatusTablet state changed. Reason: New leader 17d26ff89c1441d0991310262aa4e4e3. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.420066 11225 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:20:27.421752 10668 catalog_manager.cc:5495] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } health_report { overall_health: UNKNOWN } } }
I20260927 15:20:27.424162 11227 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:20:27.424437 11227 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:20:27.506357 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.476417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:42799 (local address 127.0.0.1:47822)
0927 15:20:27.476922 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.476945 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.477112 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.498906 (+ 21794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.498914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.498938 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:20:27.499222 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.499236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.501095 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.501103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.502836 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.502857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.504041 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.504047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.504250 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.504335 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.505501 (+ 1166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.505530 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.505612 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.505866 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260927 15:20:27.506430 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.476454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.2.246.66:42799)
0927 15:20:27.498201 (+ 21747us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.498208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.498240 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:20:27.498368 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.498374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.498523 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.498769 (+ 246us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:27.499486 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.500863 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.503153 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.503830 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.503999 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.504589 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.505377 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.505648 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.505731 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.505862 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21571,"thread_start_us":21394,"threads_started":1}
I20260927 15:20:27.513025 10832 raft_consensus.cc:1261] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Refusing update from remote peer 17d26ff89c1441d0991310262aa4e4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:27.513088 11107 raft_consensus.cc:1261] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Refusing update from remote peer 17d26ff89c1441d0991310262aa4e4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:27.514078 11225 consensus_queue.cc:1035] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:27.515012 11225 consensus_queue.cc:1035] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:27.538810 11189 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: TxnStatusTablet state changed. Reason: New leader 17d26ff89c1441d0991310262aa4e4e3. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.539321 11189 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.546551 11202 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: TxnStatusTablet state changed. Reason: New leader 17d26ff89c1441d0991310262aa4e4e3. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.546989 11202 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.557497 11188 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.557987 11233 mvcc.cc:205] Tried to move back new op lower bound from 7333979863083192320 to 7333979862702731264. Current Snapshot: MvccSnapshot[applied={T|T < 7333979863083192320}]
I20260927 15:20:27.559494 11188 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:20:27.559953 11188 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.560257 11188 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 17d26ff89c1441d0991310262aa4e4e3: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:20:27.570223 11189 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.570564 11189 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P f029dc78567f45308977558185937334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.575038 11202 tablet_replica.cc:439] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17d26ff89c1441d0991310262aa4e4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } } }
I20260927 15:20:27.575454 11202 tablet_replica.cc:442] T e8563238674c437ea034a00bf7cbd09c P 8e40a8fbb6d64d7cb1d7a829c873604d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:20:27.592902 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.585618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:40671 (local address 127.2.246.66:42799)
0927 15:20:27.586045 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.586056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.586643 (+ 587us) server_negotiation.cc:407] Connection header received
0927 15:20:27.586832 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.586837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.586928 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.587048 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:27.587640 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.588590 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.590729 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.591298 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.591458 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.592055 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.592200 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.592292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.592486 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.592582 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260927 15:20:27.593051 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.585470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:42799 (local address 127.2.246.67:40671)
0927 15:20:27.586455 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.586480 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.586670 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.587094 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.587099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.587112 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:20:27.587439 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.587450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.588757 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.588763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.590475 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.590490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.591530 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.591536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.591707 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.591789 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.592321 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.592340 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.592438 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.592594 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":620,"threads_started":1}
I20260927 15:20:27.605180 10967 raft_consensus.cc:1261] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:27.605672 10831 raft_consensus.cc:1261] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:27.605926 11202 consensus_queue.cc:1035] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:27.606531 11202 consensus_queue.cc:1035] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:27.622656 11232 mvcc.cc:205] Tried to move back new op lower bound from 7333979863464923136 to 7333979861572673536. Current Snapshot: MvccSnapshot[applied={T|T < 7333979863464923136}]
I20260927 15:20:27.650557 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.643566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40177 (local address 127.2.246.67:45251)
0927 15:20:27.643802 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.643812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.644567 (+ 755us) server_negotiation.cc:407] Connection header received
0927 15:20:27.644803 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.644809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.644889 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.644987 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:27.645561 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.646642 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.648706 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.649154 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.649290 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.649750 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.649909 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.650013 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.650147 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.650221 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:20:27.650703 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.643537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:45251 (local address 127.2.246.66:40177)
0927 15:20:27.644419 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.644436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:27.644657 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:27.645019 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:27.645024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:27.645037 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:20:27.645411 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.645419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.646797 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.646802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.648563 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:27.648577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.649370 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.649376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:27.649511 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.649569 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:20:27.650025 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:20:27.650038 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 15:20:27.650118 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:20:27.650257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":432,"threads_started":1}
I20260927 15:20:27.698606 10668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:27.698894 10668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:27.702844 10668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Sending DeleteTablet for 3 replicas of tablet d85908b45e5a4d59a6eb52e70f6a4395
I20260927 15:20:27.704113 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:27 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 15:20:27.704139 10810 tablet_service.cc:1514] Processing DeleteTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:27 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260927 15:20:27.710655 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet d85908b45e5a4d59a6eb52e70f6a4395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:27 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:47814
I20260927 15:20:27.716652 11250 tablet_replica.cc:330] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: stopping tablet replica
I20260927 15:20:27.717689 11250 raft_consensus.cc:2229] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:27.718649 11250 raft_consensus.cc:2258] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.720990 11250 ts_tablet_manager.cc:1905] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.721760 11251 tablet_replica.cc:330] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: stopping tablet replica
I20260927 15:20:27.722136 11251 raft_consensus.cc:2229] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:27.722536 11251 raft_consensus.cc:2258] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.724164 11251 ts_tablet_manager.cc:1905] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.724790 11249 tablet_replica.cc:330] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: stopping tablet replica
I20260927 15:20:27.734345 11251 ts_tablet_manager.cc:1918] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:27.734385 11249 raft_consensus.cc:2229] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:27.734683 11251 log.cc:1196] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/d85908b45e5a4d59a6eb52e70f6a4395
I20260927 15:20:27.735040 11249 raft_consensus.cc:2258] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:27.735316 11251 ts_tablet_manager.cc:1939] T d85908b45e5a4d59a6eb52e70f6a4395 P 17d26ff89c1441d0991310262aa4e4e3: Deleting consensus metadata
I20260927 15:20:27.736620 11249 ts_tablet_manager.cc:1905] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:27.736934 10656 catalog_manager.cc:4841] TS 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799): tablet d85908b45e5a4d59a6eb52e70f6a4395 (table test-table [id=94f47c19450445b7b099ff663c03ab84]) successfully deleted
I20260927 15:20:27.738248 11250 ts_tablet_manager.cc:1918] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:27.738481 11250 log.cc:1196] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/d85908b45e5a4d59a6eb52e70f6a4395
I20260927 15:20:27.738986 11250 ts_tablet_manager.cc:1939] T d85908b45e5a4d59a6eb52e70f6a4395 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting consensus metadata
I20260927 15:20:27.740494 10654 catalog_manager.cc:4841] TS 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251): tablet d85908b45e5a4d59a6eb52e70f6a4395 (table test-table [id=94f47c19450445b7b099ff663c03ab84]) successfully deleted
I20260927 15:20:27.744815 11249 ts_tablet_manager.cc:1918] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:27.745107 11249 log.cc:1196] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/d85908b45e5a4d59a6eb52e70f6a4395
I20260927 15:20:27.745592 11249 ts_tablet_manager.cc:1939] T d85908b45e5a4d59a6eb52e70f6a4395 P f029dc78567f45308977558185937334: Deleting consensus metadata
I20260927 15:20:27.747161 10657 catalog_manager.cc:4841] TS f029dc78567f45308977558185937334 (127.2.246.65:35053): tablet d85908b45e5a4d59a6eb52e70f6a4395 (table test-table [id=94f47c19450445b7b099ff663c03ab84]) successfully deleted
Sep 27 15:20:27 dist-test-slave-14jq krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:20:27 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:27 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:20:27.787792 11259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:27.773068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.2.246.126:46389)
0927 15:20:27.773676 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.773685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.774205 (+ 520us) server_negotiation.cc:407] Connection header received
0927 15:20:27.774509 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.774514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.774592 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.774729 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:27.776358 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.777398 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.779161 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.779833 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.780005 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.787460 (+ 7455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43012: BlockingRecv error: recv got EOF from 127.0.0.1:43012 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":291,"threads_started":1}
Sep 27 15:20:27 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM
I20260927 15:20:27.809412 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.791120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.2.246.126:46389)
0927 15:20:27.791328 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:27.791337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:27.791959 (+ 622us) server_negotiation.cc:407] Connection header received
0927 15:20:27.792273 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:27.792278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:27.792361 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:27.792471 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:27.794242 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.795122 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.797086 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:27.797554 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:27.797684 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:27.803047 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:27.803055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:27.803059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:27.803114 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:27.806361 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:27.807301 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:27.807307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:27.807309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:27.807485 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:27.807942 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:27.807945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:27.807947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:27.808267 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:27.808483 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:27.808881 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 15:20:27.809082 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:20:27.814441 10668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:27.815287 10668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:27.823994 10810 tablet_service.cc:1467] Processing CreateTablet for tablet 20bda0f97ee247d6b5d367f81df94208 (DEFAULT_TABLE table=test-table [id=89a4b0ce4ada423db7b3839982ae94ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.824577 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20bda0f97ee247d6b5d367f81df94208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.824595 10947 tablet_service.cc:1467] Processing CreateTablet for tablet 20bda0f97ee247d6b5d367f81df94208 (DEFAULT_TABLE table=test-table [id=89a4b0ce4ada423db7b3839982ae94ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.825184 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20bda0f97ee247d6b5d367f81df94208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.829368 11087 tablet_service.cc:1467] Processing CreateTablet for tablet 20bda0f97ee247d6b5d367f81df94208 (DEFAULT_TABLE table=test-table [id=89a4b0ce4ada423db7b3839982ae94ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:27.829864 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20bda0f97ee247d6b5d367f81df94208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:27.841382 11267 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Bootstrap starting.
I20260927 15:20:27.842401 11265 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Bootstrap starting.
I20260927 15:20:27.844419 11265 tablet_bootstrap.cc:654] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.847657 11267 tablet_bootstrap.cc:654] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.850311 11267 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: No bootstrap required, opened a new log
I20260927 15:20:27.850373 11266 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Bootstrap starting.
I20260927 15:20:27.850473 11267 ts_tablet_manager.cc:1397] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 15:20:27.851286 11267 raft_consensus.cc:348] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.851544 11267 raft_consensus.cc:374] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.851631 11267 raft_consensus.cc:729] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.851925 11267 consensus_queue.cc:260] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.852392 11266 tablet_bootstrap.cc:654] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:27.852656 11267 ts_tablet_manager.cc:1428] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:20:27.856892 11265 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: No bootstrap required, opened a new log
I20260927 15:20:27.857062 11265 ts_tablet_manager.cc:1397] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20260927 15:20:27.857882 11265 raft_consensus.cc:348] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.858181 11265 raft_consensus.cc:374] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.858286 11265 raft_consensus.cc:729] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f029dc78567f45308977558185937334, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.858548 11265 consensus_queue.cc:260] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.859158 11265 ts_tablet_manager.cc:1428] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:20:27.861778 11266 tablet_bootstrap.cc:492] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: No bootstrap required, opened a new log
I20260927 15:20:27.861989 11266 ts_tablet_manager.cc:1397] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260927 15:20:27.862922 11266 raft_consensus.cc:348] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.863195 11266 raft_consensus.cc:374] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:27.863304 11266 raft_consensus.cc:729] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Initialized, Role: FOLLOWER
I20260927 15:20:27.863626 11266 consensus_queue.cc:260] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.864249 11266 ts_tablet_manager.cc:1428] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:20:27.964757 11202 raft_consensus.cc:482] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:27.965104 11202 raft_consensus.cc:504] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.966084 11202 leader_election.cc:288] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:27.966734 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20bda0f97ee247d6b5d367f81df94208" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334" is_pre_election: true
I20260927 15:20:27.966886 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20bda0f97ee247d6b5d367f81df94208" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3" is_pre_election: true
I20260927 15:20:27.967065 10832 raft_consensus.cc:2454] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:27.967116 10967 raft_consensus.cc:2454] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:27.967710 11039 leader_election.cc:302] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d; no voters:
I20260927 15:20:27.968152 11202 raft_consensus.cc:2781] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:27.968258 11202 raft_consensus.cc:482] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:27.968385 11202 raft_consensus.cc:3037] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.970515 11202 raft_consensus.cc:504] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.971222 11202 leader_election.cc:288] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Requested vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:27.971930 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20bda0f97ee247d6b5d367f81df94208" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334"
I20260927 15:20:27.972064 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20bda0f97ee247d6b5d367f81df94208" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3"
I20260927 15:20:27.972177 10832 raft_consensus.cc:3037] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.972275 10967 raft_consensus.cc:3037] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:27.975267 10967 raft_consensus.cc:2454] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:27.975267 10832 raft_consensus.cc:2454] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:27.975831 11040 leader_election.cc:302] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e40a8fbb6d64d7cb1d7a829c873604d, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:27.976241 11202 raft_consensus.cc:2781] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:27.976529 11202 raft_consensus.cc:686] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Becoming Leader. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Running, Role: LEADER
I20260927 15:20:27.976814 11202 consensus_queue.cc:237] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:27.979879 10670 catalog_manager.cc:5495] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67). New cstate: current_term: 1 leader_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } health_report { overall_health: HEALTHY } } }
I20260927 15:20:28.014819 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.009583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.2.246.67:45251)
0927 15:20:28.009757 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:28.009762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:28.009957 (+ 195us) server_negotiation.cc:407] Connection header received
0927 15:20:28.010112 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:28.010116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:28.010184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:28.010275 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:28.010651 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.011366 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.012853 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.013285 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.013392 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:28.014059 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:28.014175 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:28.014257 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:28.014455 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 15:20:28.014521 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:20:28.018095 10967 raft_consensus.cc:1261] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:28.018355 10832 raft_consensus.cc:1261] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:28.018813 11202 consensus_queue.cc:1035] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:28.019099 11193 consensus_queue.cc:1035] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:28.034049 10670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:28.034364 10670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:28.037950 10670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Sending DeleteTablet for 3 replicas of tablet 20bda0f97ee247d6b5d367f81df94208
I20260927 15:20:28.039144 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet 20bda0f97ee247d6b5d367f81df94208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 15:20:28.039152 10810 tablet_service.cc:1514] Processing DeleteTablet for tablet 20bda0f97ee247d6b5d367f81df94208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260927 15:20:28.039229 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet 20bda0f97ee247d6b5d367f81df94208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:47814
I20260927 15:20:28.039614 11249 tablet_replica.cc:330] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: stopping tablet replica
I20260927 15:20:28.039613 11250 tablet_replica.cc:330] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: stopping tablet replica
I20260927 15:20:28.039873 11249 raft_consensus.cc:2229] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:28.039915 11251 tablet_replica.cc:330] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: stopping tablet replica
I20260927 15:20:28.039922 11250 raft_consensus.cc:2229] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:28.040124 11251 raft_consensus.cc:2229] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:28.040162 11249 raft_consensus.cc:2258] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.040364 11251 raft_consensus.cc:2258] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.040364 11250 raft_consensus.cc:2258] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.040700 10670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:20:28.041427 11249 ts_tablet_manager.cc:1905] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:20:28.041664 10670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:28.041711 11250 ts_tablet_manager.cc:1905] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:28.041923 11251 ts_tablet_manager.cc:1905] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:28.048646 11250 ts_tablet_manager.cc:1918] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:28.048826 11250 log.cc:1196] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/20bda0f97ee247d6b5d367f81df94208
I20260927 15:20:28.048866 11249 ts_tablet_manager.cc:1918] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:28.048866 11251 ts_tablet_manager.cc:1918] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:28.049048 11249 log.cc:1196] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/20bda0f97ee247d6b5d367f81df94208
I20260927 15:20:28.049060 11251 log.cc:1196] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/20bda0f97ee247d6b5d367f81df94208
I20260927 15:20:28.049194 11250 ts_tablet_manager.cc:1939] T 20bda0f97ee247d6b5d367f81df94208 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting consensus metadata
I20260927 15:20:28.049518 11249 ts_tablet_manager.cc:1939] T 20bda0f97ee247d6b5d367f81df94208 P f029dc78567f45308977558185937334: Deleting consensus metadata
I20260927 15:20:28.049530 11251 ts_tablet_manager.cc:1939] T 20bda0f97ee247d6b5d367f81df94208 P 17d26ff89c1441d0991310262aa4e4e3: Deleting consensus metadata
I20260927 15:20:28.050876 10657 catalog_manager.cc:4841] TS f029dc78567f45308977558185937334 (127.2.246.65:35053): tablet 20bda0f97ee247d6b5d367f81df94208 (table test-table [id=89a4b0ce4ada423db7b3839982ae94ff]) successfully deleted
I20260927 15:20:28.051367 10654 catalog_manager.cc:4841] TS 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251): tablet 20bda0f97ee247d6b5d367f81df94208 (table test-table [id=89a4b0ce4ada423db7b3839982ae94ff]) successfully deleted
I20260927 15:20:28.051609 10656 catalog_manager.cc:4841] TS 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799): tablet 20bda0f97ee247d6b5d367f81df94208 (table test-table [id=89a4b0ce4ada423db7b3839982ae94ff]) successfully deleted
I20260927 15:20:28.057992 10810 tablet_service.cc:1467] Processing CreateTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 (DEFAULT_TABLE table=test-table [id=3c391821cb8e45c5a1aee73ac9823be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:28.058339 11087 tablet_service.cc:1467] Processing CreateTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 (DEFAULT_TABLE table=test-table [id=3c391821cb8e45c5a1aee73ac9823be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:28.058470 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab53030c4b3d46a0b862918c8d15f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:28.058408 10947 tablet_service.cc:1467] Processing CreateTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 (DEFAULT_TABLE table=test-table [id=3c391821cb8e45c5a1aee73ac9823be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:28.058749 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab53030c4b3d46a0b862918c8d15f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:28.059000 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab53030c4b3d46a0b862918c8d15f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:28.065037 11267 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Bootstrap starting.
I20260927 15:20:28.065109 11266 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Bootstrap starting.
I20260927 15:20:28.066572 11267 tablet_bootstrap.cc:654] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:28.067112 11266 tablet_bootstrap.cc:654] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:28.067855 11265 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Bootstrap starting.
I20260927 15:20:28.068750 11267 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: No bootstrap required, opened a new log
I20260927 15:20:28.068930 11267 ts_tablet_manager.cc:1397] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 15:20:28.069557 11267 raft_consensus.cc:348] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.069787 11267 raft_consensus.cc:374] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:28.069828 11266 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: No bootstrap required, opened a new log
I20260927 15:20:28.069898 11265 tablet_bootstrap.cc:654] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:28.069912 11267 raft_consensus.cc:729] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Initialized, Role: FOLLOWER
I20260927 15:20:28.070039 11266 ts_tablet_manager.cc:1397] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 15:20:28.070191 11267 consensus_queue.cc:260] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.070948 11266 raft_consensus.cc:348] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.071157 11267 ts_tablet_manager.cc:1428] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:20:28.071278 11266 raft_consensus.cc:374] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:28.071380 11266 raft_consensus.cc:729] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d26ff89c1441d0991310262aa4e4e3, State: Initialized, Role: FOLLOWER
I20260927 15:20:28.071712 11266 consensus_queue.cc:260] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.072932 11265 tablet_bootstrap.cc:492] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: No bootstrap required, opened a new log
I20260927 15:20:28.073143 11265 ts_tablet_manager.cc:1397] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 15:20:28.073432 11266 ts_tablet_manager.cc:1428] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:28.074155 11265 raft_consensus.cc:348] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.074447 11265 raft_consensus.cc:374] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:28.074532 11265 raft_consensus.cc:729] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f029dc78567f45308977558185937334, State: Initialized, Role: FOLLOWER
I20260927 15:20:28.074769 11265 consensus_queue.cc:260] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.075352 11265 ts_tablet_manager.cc:1428] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:28.180014 11202 raft_consensus.cc:482] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:28.180444 11202 raft_consensus.cc:504] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.181227 11202 leader_election.cc:288] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:28.182144 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab53030c4b3d46a0b862918c8d15f8d3" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334" is_pre_election: true
I20260927 15:20:28.182144 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab53030c4b3d46a0b862918c8d15f8d3" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3" is_pre_election: true
I20260927 15:20:28.182461 10966 raft_consensus.cc:2454] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:28.182498 10832 raft_consensus.cc:2454] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 0.
I20260927 15:20:28.183118 11040 leader_election.cc:302] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17d26ff89c1441d0991310262aa4e4e3, 8e40a8fbb6d64d7cb1d7a829c873604d, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:28.183604 11202 raft_consensus.cc:2781] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:28.183763 11202 raft_consensus.cc:482] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:28.183894 11202 raft_consensus.cc:3037] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:28.186144 11202 raft_consensus.cc:504] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.186856 11202 leader_election.cc:288] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Requested vote from peers f029dc78567f45308977558185937334 (127.2.246.65:35053), 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799)
I20260927 15:20:28.187665 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab53030c4b3d46a0b862918c8d15f8d3" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f029dc78567f45308977558185937334"
I20260927 15:20:28.187667 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab53030c4b3d46a0b862918c8d15f8d3" candidate_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d26ff89c1441d0991310262aa4e4e3"
I20260927 15:20:28.187950 10966 raft_consensus.cc:3037] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:28.187951 10832 raft_consensus.cc:3037] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:28.190474 10832 raft_consensus.cc:2454] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:28.190568 10966 raft_consensus.cc:2454] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e40a8fbb6d64d7cb1d7a829c873604d in term 1.
I20260927 15:20:28.190991 11040 leader_election.cc:302] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e40a8fbb6d64d7cb1d7a829c873604d, f029dc78567f45308977558185937334; no voters:
I20260927 15:20:28.191356 11202 raft_consensus.cc:2781] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:28.191532 11202 raft_consensus.cc:686] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Becoming Leader. State: Replica: 8e40a8fbb6d64d7cb1d7a829c873604d, State: Running, Role: LEADER
I20260927 15:20:28.191787 11202 consensus_queue.cc:237] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } }
I20260927 15:20:28.194842 10670 catalog_manager.cc:5495] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67). New cstate: current_term: 1 leader_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e40a8fbb6d64d7cb1d7a829c873604d" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45251 } health_report { overall_health: HEALTHY } } }
I20260927 15:20:28.251191 10966 raft_consensus.cc:1261] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:28.251302 10831 raft_consensus.cc:1261] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Refusing update from remote peer 8e40a8fbb6d64d7cb1d7a829c873604d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:28.251935 11202 consensus_queue.cc:1035] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f029dc78567f45308977558185937334" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:28.252240 11244 consensus_queue.cc:1035] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d26ff89c1441d0991310262aa4e4e3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:28.316146 10668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:28.316497 10668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:28.320115 10668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dfeba54145e4d05ba06cf6877760543: Sending DeleteTablet for 3 replicas of tablet ab53030c4b3d46a0b862918c8d15f8d3
I20260927 15:20:28.321192 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:47814
I20260927 15:20:28.321218 10810 tablet_service.cc:1514] Processing DeleteTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260927 15:20:28.321643 11249 tablet_replica.cc:330] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: stopping tablet replica
I20260927 15:20:28.321674 11251 tablet_replica.cc:330] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: stopping tablet replica
I20260927 15:20:28.321858 11249 raft_consensus.cc:2229] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:28.321887 11087 tablet_service.cc:1514] Processing DeleteTablet for tablet ab53030c4b3d46a0b862918c8d15f8d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:28 UTC) from {username='oryx', principal='oryx/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 15:20:28.322084 11249 raft_consensus.cc:2258] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.322271 11250 tablet_replica.cc:330] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: stopping tablet replica
I20260927 15:20:28.322515 11250 raft_consensus.cc:2229] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:28.322645 11251 raft_consensus.cc:2229] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:28.322950 11250 raft_consensus.cc:2258] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.322950 11251 raft_consensus.cc:2258] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:28.323544 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 10741
I20260927 15:20:28.324111 11249 ts_tablet_manager.cc:1905] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:28.324358 11251 ts_tablet_manager.cc:1905] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:28.324771 11250 ts_tablet_manager.cc:1905] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:28.332060 11250 ts_tablet_manager.cc:1918] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:28.332064 11249 ts_tablet_manager.cc:1918] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:28.332249 11249 log.cc:1196] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/ab53030c4b3d46a0b862918c8d15f8d3
I20260927 15:20:28.332432 11250 log.cc:1196] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/ab53030c4b3d46a0b862918c8d15f8d3
I20260927 15:20:28.332967 11249 ts_tablet_manager.cc:1939] T ab53030c4b3d46a0b862918c8d15f8d3 P f029dc78567f45308977558185937334: Deleting consensus metadata
I20260927 15:20:28.333196 11250 ts_tablet_manager.cc:1939] T ab53030c4b3d46a0b862918c8d15f8d3 P 8e40a8fbb6d64d7cb1d7a829c873604d: Deleting consensus metadata
I20260927 15:20:28.334769 10657 catalog_manager.cc:4841] TS f029dc78567f45308977558185937334 (127.2.246.65:35053): tablet ab53030c4b3d46a0b862918c8d15f8d3 (table test-table [id=3c391821cb8e45c5a1aee73ac9823be9]) successfully deleted
I20260927 15:20:28.334838 10654 catalog_manager.cc:4841] TS 8e40a8fbb6d64d7cb1d7a829c873604d (127.2.246.67:45251): tablet ab53030c4b3d46a0b862918c8d15f8d3 (table test-table [id=3c391821cb8e45c5a1aee73ac9823be9]) successfully deleted
I20260927 15:20:28.336916 11251 ts_tablet_manager.cc:1918] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:20:28.337090 11251 log.cc:1196] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/ab53030c4b3d46a0b862918c8d15f8d3
I20260927 15:20:28.337596 11251 ts_tablet_manager.cc:1939] T ab53030c4b3d46a0b862918c8d15f8d3 P 17d26ff89c1441d0991310262aa4e4e3: Deleting consensus metadata
I20260927 15:20:28.338840 10656 catalog_manager.cc:4841] TS 17d26ff89c1441d0991310262aa4e4e3 (127.2.246.66:42799): tablet ab53030c4b3d46a0b862918c8d15f8d3 (table test-table [id=3c391821cb8e45c5a1aee73ac9823be9]) successfully deleted
Sep 27 15:20:28 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.65@KRBTEST.COM
I20260927 15:20:28.353130 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.330101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.2.246.65:35053)
0927 15:20:28.335918 (+ 5817us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:28.335923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:28.335943 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:28.336001 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:28.336006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:28.336096 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:28.336245 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:28.338258 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.339313 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.340942 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.341431 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.341589 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:28.346547 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:28.346574 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:28.346599 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:28.346645 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:28.349782 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.350754 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.350761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.350764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.350943 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.351451 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.351455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.351458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.351848 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:28.352096 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:28.352501 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 15:20:28.352696 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5657,"thread_start_us":5507,"threads_started":1}
I20260927 15:20:28.353937 10872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:28.458994 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 10741
I20260927 15:20:28.483506 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 10881
Sep 27 15:20:28 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.66@KRBTEST.COM
I20260927 15:20:28.502236 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.484651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.2.246.66:42799)
0927 15:20:28.485379 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:28.485384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:28.485403 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:20:28.485527 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:28.485535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:28.485620 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:28.485759 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:28.487401 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.488388 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.490012 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.490450 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.490573 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:28.495116 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:28.495137 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:28.495161 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:28.495212 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:28.498853 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.499659 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.499669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.499672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.499965 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.500371 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.500375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.500378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.500975 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:28.501321 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:28.501516 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:20:28.501762 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":387,"threads_started":1}
I20260927 15:20:28.503116 11012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:28.630646 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 10881
I20260927 15:20:28.657429 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11021
Sep 27 15:20:28 dist-test-slave-14jq krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.246.67@KRBTEST.COM
I20260927 15:20:28.676561 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.658516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.2.246.67:45251)
0927 15:20:28.659267 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:28.659272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:28.659294 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:28.659368 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:28.659373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:28.659457 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:28.659577 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:28.660886 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.662122 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.663479 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:28.663962 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:28.664103 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:28.668901 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:28.668922 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:28.668943 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:28.668994 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:28.672221 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.673091 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.673099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.673101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.673244 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:28.674329 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:28.674337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:28.674340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:28.675114 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:28.675571 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:28.675874 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:20:28.676155 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":484,"threads_started":1}
I20260927 15:20:28.677465 11152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:28.820591 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11021
I20260927 15:20:28.849244 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 10638
I20260927 15:20:28.851234 10710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:28.973176 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 10638
2026-09-27T15:20:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7520 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:29 dist-test-slave-14jq krb5kdc[11296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:29 dist-test-slave-14jq krb5kdc[11296](info): set up 2 sockets
Sep 27 15:20:29 dist-test-slave-14jq krb5kdc[11296](info): commencing operation
krb5kdc: starting...
W20260927 15:20:30.106914 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:20:30 dist-test-slave-14jq krb5kdc[11296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:20:30.407758 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35777
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:30.836081 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:30.836503 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:30.836691 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:30.844805 11309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:30.844993 11309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:30.845104 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:30.845224 11309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:30.845301 11309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:30.855419 11309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:30.856900 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:30.858359 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:30.872254 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:30.872346 11317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:30.872346 11315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:30.873992 11309 server_base.cc:1034] running on GCE node
I20260927 15:20:30.875113 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:30.877071 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:30.878347 11309 hybrid_clock.cc:648] HybridClock initialized: now 1790522430878251 us; error 69 us; skew 500 ppm
I20260927 15:20:30.878909 11309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:20:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1973 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:31 dist-test-slave-14jq krb5kdc[11323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:31 dist-test-slave-14jq krb5kdc[11323](info): set up 2 sockets
Sep 27 15:20:31 dist-test-slave-14jq krb5kdc[11323](info): commencing operation
krb5kdc: starting...
W20260927 15:20:32.122998 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:20:32 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:20:32.407305 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33307
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:32.845024 11336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:32.845420 11336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:32.845646 11336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:32.854535 11336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:32.854719 11336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:32.854838 11336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:32.854943 11336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:32.855015 11336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:32.866611 11336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:32.868500 11336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:32.870656 11336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:32.887351 11341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.887370 11342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.887382 11344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.888592 11336 server_base.cc:1034] running on GCE node
I20260927 15:20:32.889760 11336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:32.892091 11336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:32.893406 11336 hybrid_clock.cc:648] HybridClock initialized: now 1790522432893275 us; error 107 us; skew 500 ppm
I20260927 15:20:32.894033 11336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:32 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:32.897953 11336 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:20:32.899909 11336 webserver.cc:460] Webserver started at http://127.2.246.126:38889/ using document root <none> and password file <none>
I20260927 15:20:32.900838 11336 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:32.900988 11336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:32.901356 11336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:32.905645 11336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "d43385ffce2e4669b21fc1be17646000"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-14jq"
server_key: "16b1eed4a2b88a387668c5d4036f315c"
server_key_iv: "f25264a331eb645acd48d5f18b8abd4a"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.907181 11336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "d43385ffce2e4669b21fc1be17646000"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-14jq"
server_key: "16b1eed4a2b88a387668c5d4036f315c"
server_key_iv: "f25264a331eb645acd48d5f18b8abd4a"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.915608 11336 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 15:20:32.921154 11351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.923465 11336 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:20:32.923782 11336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "d43385ffce2e4669b21fc1be17646000"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-14jq"
server_key: "16b1eed4a2b88a387668c5d4036f315c"
server_key_iv: "f25264a331eb645acd48d5f18b8abd4a"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.924063 11336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:32.941548 11336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:32.948130 11336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:32.948531 11336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:32.972357 11336 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33307
I20260927 15:20:32.972386 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33307 every 8 connection(s)
I20260927 15:20:32.974287 11336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:32.976557 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11336
I20260927 15:20:32.976821 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:32.977710 3033 external_mini_cluster.cc:1428] Setting key 3c9bc4fe8892a0125c42effe29451b76
I20260927 15:20:32.980058 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:20:32 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:32.995990 11403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Bootstrap starting.
I20260927 15:20:33.000564 11403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:33.002050 11403 log.cc:824] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:33.006760 11403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: No bootstrap required, opened a new log
I20260927 15:20:33.008164 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.980441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.2.246.126:33307)
0927 15:20:32.981477 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.981492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.981547 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:20:32.983478 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.983506 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.984064 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.984517 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.986564 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.988190 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.990248 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.991178 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.991383 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.996879 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.996893 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.996915 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:32.996979 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:33.001635 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:33.003209 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:33.003217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:33.003224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:33.003468 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:33.004199 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:33.004206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:33.004210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:33.005073 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:33.005422 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:33.006074 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 15:20:33.006587 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":465,"threads_started":1}
I20260927 15:20:33.016508 11403 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:33.017081 11403 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:33.017251 11403 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43385ffce2e4669b21fc1be17646000, State: Initialized, Role: FOLLOWER
I20260927 15:20:33.018059 11403 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:33.018285 11403 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:33.018447 11403 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:33.018651 11403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:33.020844 11403 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:33.021396 11403 leader_election.cc:302] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43385ffce2e4669b21fc1be17646000; no voters:
I20260927 15:20:33.022190 11403 leader_election.cc:288] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:33.022485 11408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:33.024116 11408 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 LEADER]: Becoming Leader. State: Replica: d43385ffce2e4669b21fc1be17646000, State: Running, Role: LEADER
I20260927 15:20:33.024731 11403 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:33.024998 11408 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:33.030190 11409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43385ffce2e4669b21fc1be17646000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } } }
I20260927 15:20:33.030812 11409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:33.030196 11410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43385ffce2e4669b21fc1be17646000. Latest consensus state: current_term: 1 leader_uuid: "d43385ffce2e4669b21fc1be17646000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } } }
I20260927 15:20:33.030953 11410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:33.034497 11419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:33.042268 11419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:33.054358 11419 catalog_manager.cc:1347] Generated new cluster ID: 5f87b6d2647a417cbb229204578ed798
I20260927 15:20:33.054536 11419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:33.079145 11419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:33.081387 11419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:33.089334 11419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Generated new TSK 0
I20260927 15:20:33.090641 11419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:20:33.166507 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:33.623382 11429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:33.623835 11429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:33.624038 11429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:33.633868 11429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:33.634047 11429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:33.634265 11429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:33.648675 11429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:33.650755 11429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:33.652690 11429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:33.670078 11434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:33.670109 11435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:33.670122 11437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:33.671530 11429 server_base.cc:1034] running on GCE node
I20260927 15:20:33.672622 11429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:33.674031 11429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:33.675308 11429 hybrid_clock.cc:648] HybridClock initialized: now 1790522433675256 us; error 46 us; skew 500 ppm
I20260927 15:20:33.675743 11429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:33 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:33.680060 11429 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:20:33.682101 11429 webserver.cc:460] Webserver started at http://127.2.246.65:41859/ using document root <none> and password file <none>
I20260927 15:20:33.683007 11429 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:33.683130 11429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:33.683480 11429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:33.687278 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "d75760c8223546c2b6bf48a6f27e5758"
format_stamp: "Formatted at 2026-09-27 15:20:33 on dist-test-slave-14jq"
server_key: "9283801a9c3c8c5807058952a417bb0e"
server_key_iv: "f1bdaa1db1a54be5a87f60d529de6223"
server_key_version: "encryptionkey@0"
I20260927 15:20:33.688329 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "d75760c8223546c2b6bf48a6f27e5758"
format_stamp: "Formatted at 2026-09-27 15:20:33 on dist-test-slave-14jq"
server_key: "9283801a9c3c8c5807058952a417bb0e"
server_key_iv: "f1bdaa1db1a54be5a87f60d529de6223"
server_key_version: "encryptionkey@0"
I20260927 15:20:33.695390 11429 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260927 15:20:33.700799 11444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:33.702965 11429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:33.703187 11429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "d75760c8223546c2b6bf48a6f27e5758"
format_stamp: "Formatted at 2026-09-27 15:20:33 on dist-test-slave-14jq"
server_key: "9283801a9c3c8c5807058952a417bb0e"
server_key_iv: "f1bdaa1db1a54be5a87f60d529de6223"
server_key_version: "encryptionkey@0"
I20260927 15:20:33.703472 11429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:33.736366 11429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:33.742103 11429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:33.742424 11429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:33.743577 11429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:33.745359 11429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:33.745494 11429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:33.745663 11429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:33.745747 11429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:33.785262 11429 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:37517
I20260927 15:20:33.785329 11556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:37517 every 8 connection(s)
I20260927 15:20:33.786980 11429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:33.789875 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11429
I20260927 15:20:33.790132 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:33.790973 3033 external_mini_cluster.cc:1428] Setting key b8a9aa30b616a6722d2fa3788e3d9124
Sep 27 15:20:33 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:33.811888 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:33.790395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:49245 (local address 127.2.246.126:33307)
0927 15:20:33.790976 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:33.790983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:33.792421 (+ 1438us) server_negotiation.cc:407] Connection header received
0927 15:20:33.794645 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:33.794651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:33.794743 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:33.794868 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:33.796466 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.797191 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.799205 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.800028 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.800247 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:33.805093 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:33.805099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:33.805102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:33.805145 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:33.808123 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:33.809183 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:33.809189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:33.809191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:33.809329 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:33.809819 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:33.809823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:33.809825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:33.810170 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:33.810330 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:33.811198 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 15:20:33.811481 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":275,"threads_started":1}
I20260927 15:20:33.813148 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:33.791296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:49245)
0927 15:20:33.792230 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:33.792283 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:33.794345 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:33.795126 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:33.795135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:33.795687 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:33.796292 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:33.796310 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.797362 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.797367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:33.798971 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:33.798988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.800331 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.800340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:33.800669 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:33.801602 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:33.801638 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:33.804776 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:33.808333 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:33.808346 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:33.808373 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:33.809031 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:33.809447 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:33.809453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:33.809457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:33.809682 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:33.810306 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:33.810316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:33.810780 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 15:20:33.811089 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 15:20:33.811767 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":397,"threads_started":1}
I20260927 15:20:33.815258 11557 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33307
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:20:33.815663 11557 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:33.816558 11557 heartbeater.cc:511] Master 127.2.246.126:33307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:20:33.819393 11368 ts_manager.cc:194] Registered new tserver with Master: d75760c8223546c2b6bf48a6f27e5758 (127.2.246.65:37517)
I20260927 15:20:33.821120 11368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:49245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:20:33.849216 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:34.214653 11563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.215096 11563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.215276 11563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.223785 11563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.223997 11563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.224196 11563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:34.236372 11563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:34.238302 11563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.240159 11563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:34.258078 11571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.258373 11568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.258387 11569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.259843 11563 server_base.cc:1034] running on GCE node
I20260927 15:20:34.260764 11563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.262151 11563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.263473 11563 hybrid_clock.cc:648] HybridClock initialized: now 1790522434263373 us; error 92 us; skew 500 ppm
I20260927 15:20:34.263922 11563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:34 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.268210 11563 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:20:34.270296 11563 webserver.cc:460] Webserver started at http://127.2.246.66:34243/ using document root <none> and password file <none>
I20260927 15:20:34.271311 11563 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.271438 11563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.271759 11563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:34.276075 11563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "8306ebb409784e93ae07998b57c43448"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "d51f913288df4b698bfbac859e51df60"
server_key_iv: "a216bcf816486c0913fd84d34f63b860"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.277241 11563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "8306ebb409784e93ae07998b57c43448"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "d51f913288df4b698bfbac859e51df60"
server_key_iv: "a216bcf816486c0913fd84d34f63b860"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.287621 11563 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.003s
I20260927 15:20:34.292881 11578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.295070 11563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:34.295365 11563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "8306ebb409784e93ae07998b57c43448"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "d51f913288df4b698bfbac859e51df60"
server_key_iv: "a216bcf816486c0913fd84d34f63b860"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.295651 11563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:34.350126 11563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:34.356525 11563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:34.356925 11563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:34.358573 11563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:34.360759 11563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:34.360875 11563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.361038 11563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:34.361094 11563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.403788 11563 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:38735
I20260927 15:20:34.403811 11690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:38735 every 8 connection(s)
I20260927 15:20:34.405582 11563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:34.407371 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11563
I20260927 15:20:34.407581 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:34.408212 3033 external_mini_cluster.cc:1428] Setting key ff35bb18a2f56143a1d186afb47bf54a
Sep 27 15:20:34 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:34.436054 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.409515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:43677 (local address 127.2.246.126:33307)
0927 15:20:34.410061 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:34.410065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:34.411808 (+ 1743us) server_negotiation.cc:407] Connection header received
0927 15:20:34.415017 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:34.415023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:34.415135 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:34.415290 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:34.417740 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.418840 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.420984 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.421941 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.422146 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.426840 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:34.426851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:34.426860 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:34.426928 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:34.432060 (+ 5132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.433176 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.433184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.433190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.433439 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.434100 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.434107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.434109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.434477 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:34.434647 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:34.435374 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 15:20:34.435647 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":270,"threads_started":1}
I20260927 15:20:34.437223 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.410328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:43677)
0927 15:20:34.411596 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:34.411666 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:34.414607 (+ 2941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:34.415634 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:34.415649 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:34.416529 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:34.417449 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.417481 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.419191 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.419196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.420732 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.420746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.422256 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.422262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.422558 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.423487 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:34.423517 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:34.426442 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:34.432354 (+ 5912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.432364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.432380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.432972 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.433638 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.433642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.433644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.433911 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.434629 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:34.434638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:34.435009 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 15:20:34.435279 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 15:20:34.435876 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":495,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:20:34.439064 11691 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33307
I20260927 15:20:34.439528 11691 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:34.440485 11691 heartbeater.cc:511] Master 127.2.246.126:33307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:20:34.442658 11368 ts_manager.cc:194] Registered new tserver with Master: 8306ebb409784e93ae07998b57c43448 (127.2.246.66:38735)
I20260927 15:20:34.443708 11368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:43677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:20:34.473789 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:34.825670 11557 heartbeater.cc:503] Master 127.2.246.126:33307 was elected leader, sending a full tablet report...
W20260927 15:20:34.901408 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.901937 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.902140 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.910946 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.911140 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.911334 11697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:34.922508 11697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:34.924319 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.926120 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:34.942200 11702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.942308 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.942927 11705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.943863 11697 server_base.cc:1034] running on GCE node
I20260927 15:20:34.944826 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.946185 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.947450 11697 hybrid_clock.cc:648] HybridClock initialized: now 1790522434947380 us; error 62 us; skew 500 ppm
I20260927 15:20:34.947907 11697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:34 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.951990 11697 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:20:34.954140 11697 webserver.cc:460] Webserver started at http://127.2.246.67:37579/ using document root <none> and password file <none>
I20260927 15:20:34.955080 11697 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.955237 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.955632 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:34.959503 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "337be0d736a447fd90b7ff0308f6e176"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "b44443559717d03e4c4f6e348168c1e7"
server_key_iv: "24c389bbda98054c55253f0bd53b12d9"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.960548 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "337be0d736a447fd90b7ff0308f6e176"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "b44443559717d03e4c4f6e348168c1e7"
server_key_iv: "24c389bbda98054c55253f0bd53b12d9"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.967753 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 15:20:34.973956 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.976172 11697 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:20:34.976469 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "337be0d736a447fd90b7ff0308f6e176"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "b44443559717d03e4c4f6e348168c1e7"
server_key_iv: "24c389bbda98054c55253f0bd53b12d9"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.976748 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:35.005671 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:35.012054 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:35.012431 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:35.013674 11697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:35.015813 11697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:35.016008 11697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.016181 11697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:35.016258 11697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.059923 11697 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:40407
I20260927 15:20:35.059974 11824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:40407 every 8 connection(s)
I20260927 15:20:35.061717 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:35.063608 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11697
I20260927 15:20:35.063793 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:35.064347 3033 external_mini_cluster.cc:1428] Setting key 9e6e697fbd3dfa146665441eab42ebcd
Sep 27 15:20:35 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:20:35.087867 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.065150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:51019 (local address 127.2.246.126:33307)
0927 15:20:35.065649 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.065655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.067415 (+ 1760us) server_negotiation.cc:407] Connection header received
0927 15:20:35.069570 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.069575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.069665 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.069815 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.071742 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.072657 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.074790 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.075292 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.075456 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.080023 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.080029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.080033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.080120 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.083911 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.084972 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.084977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.084980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.085156 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.085787 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.085791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.085794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.086263 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.086409 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.087252 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.087493 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":539,"threads_started":1}
I20260927 15:20:35.088713 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.066002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:51019)
0927 15:20:35.067165 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.067234 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.069229 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.070171 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.070181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.070761 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:35.071387 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.071405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.072902 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.072906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.074553 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.074569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.075663 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.075669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.075902 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.076707 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:35.076732 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:35.079675 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:35.084128 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:35.084137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:35.084152 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:35.084775 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:35.085315 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:35.085320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:35.085324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:35.085596 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:35.086437 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:35.086446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:35.086846 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.087111 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.087548 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":463,"threads_started":1}
I20260927 15:20:35.090845 11825 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33307
I20260927 15:20:35.091218 11825 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:35.092212 11825 heartbeater.cc:511] Master 127.2.246.126:33307 requested a full tablet report, sending...
I20260927 15:20:35.094487 11367 ts_manager.cc:194] Registered new tserver with Master: 337be0d736a447fd90b7ff0308f6e176 (127.2.246.67:40407)
I20260927 15:20:35.095821 11367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:51019
I20260927 15:20:35.108649 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
Entry for principal oryx/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab.
WARNING: no policy specified for oryx/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
Entry for principal oryx/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab.
WARNING: no policy specified for oryx/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
Entry for principal oryx/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab.
I20260927 15:20:35.269183 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11429
Sep 27 15:20:35 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:20:35.289523 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.270796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.2.246.65:37517)
0927 15:20:35.271711 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.271721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.272082 (+ 361us) server_negotiation.cc:407] Connection header received
0927 15:20:35.272238 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.272246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.272554 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.272746 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.274332 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.275529 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.277048 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.277924 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.278087 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.282798 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.282820 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.282841 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.282896 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.286459 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.287328 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.287346 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.287348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.287502 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.287928 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.287931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.287933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.288349 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.288604 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.288925 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.289169 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":414,"threads_started":1}
I20260927 15:20:35.291320 11552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:35.381186 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11429
I20260927 15:20:35.402247 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11563
Sep 27 15:20:35 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:20:35.421895 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.403234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.2.246.66:38735)
0927 15:20:35.403928 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.403936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.403964 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:20:35.404040 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.404048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.404276 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.404415 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.406389 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.407656 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.409743 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.410534 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.410662 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.415722 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.415736 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.415753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.415796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.418898 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.419704 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.419712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.419714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.419834 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.420203 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.420220 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.420223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.420611 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.420927 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.421228 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.421490 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":405,"threads_started":1}
I20260927 15:20:35.423388 11686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:35.515168 11691 heartbeater.cc:503] Master 127.2.246.126:33307 was elected leader, sending a full tablet report...
I20260927 15:20:35.517040 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11563
W20260927 15:20:35.535653 11354 connection.cc:535] server connection from 127.2.246.66:43677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:20:35.536036 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11697
Sep 27 15:20:35 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:20:35.557407 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.537126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.2.246.67:40407)
0927 15:20:35.538335 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.538345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.538401 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:20:35.538495 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.538503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.538793 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.538977 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.540465 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.541593 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.543243 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.544320 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.544473 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.549982 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.550006 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.550028 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.550086 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.553725 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.554772 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.554813 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.554820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.555027 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.555483 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.555487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.555490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.555989 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.556268 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.556621 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.556997 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":848,"threads_started":1}
I20260927 15:20:35.559036 11820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:35.647650 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11697
I20260927 15:20:35.664119 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11336
I20260927 15:20:35.665674 11398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:35.787331 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11336
I20260927 15:20:35.810688 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33307
--webserver_interface=127.2.246.126
--webserver_port=38889
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:36.197343 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:36.197667 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:36.197798 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:36.205585 11843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:36.205711 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:36.205758 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:36.205838 11843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:36.205953 11843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:36.216713 11843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=38889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:36.218211 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:36.219630 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:36.234139 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.234483 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.234694 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:36.235958 11843 server_base.cc:1034] running on GCE node
I20260927 15:20:36.237129 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:36.239142 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:36.240460 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790522436240364 us; error 85 us; skew 500 ppm
I20260927 15:20:36.240983 11843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:36 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:36.245217 11843 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:20:36.247990 11843 webserver.cc:460] Webserver started at http://127.2.246.126:38889/ using document root <none> and password file <none>
I20260927 15:20:36.249150 11843 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:36.249547 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:36.257385 11843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 15:20:36.261112 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.263123 11843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260927 15:20:36.263355 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "d43385ffce2e4669b21fc1be17646000"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-14jq"
server_key: "16b1eed4a2b88a387668c5d4036f315c"
server_key_iv: "f25264a331eb645acd48d5f18b8abd4a"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.264493 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:36.292834 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:36.297801 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:36.298157 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:36.319911 11843 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33307
I20260927 15:20:36.320034 11909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33307 every 8 connection(s)
I20260927 15:20:36.321605 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:36.328784 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11843
I20260927 15:20:36.329927 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:37517
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:33307
--webserver_port=41859
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:36.331415 11910 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:20:36.337747 11910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Bootstrap starting.
I20260927 15:20:36.363142 11910 log.cc:824] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:36.375594 11910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:20:36.376395 11910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Bootstrap complete.
I20260927 15:20:36.388069 11910 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:36.389402 11910 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43385ffce2e4669b21fc1be17646000, State: Initialized, Role: FOLLOWER
I20260927 15:20:36.390453 11910 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:36.390777 11910 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:36.390921 11910 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:36.391161 11910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:20:36.397034 11910 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:36.398160 11910 leader_election.cc:302] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43385ffce2e4669b21fc1be17646000; no voters:
I20260927 15:20:36.399115 11910 leader_election.cc:288] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:20:36.399317 11914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:20:36.400861 11914 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [term 2 LEADER]: Becoming Leader. State: Replica: d43385ffce2e4669b21fc1be17646000, State: Running, Role: LEADER
I20260927 15:20:36.401738 11914 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } }
I20260927 15:20:36.402289 11910 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:36.405439 11916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43385ffce2e4669b21fc1be17646000. Latest consensus state: current_term: 2 leader_uuid: "d43385ffce2e4669b21fc1be17646000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } } }
I20260927 15:20:36.406054 11916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:36.405539 11915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d43385ffce2e4669b21fc1be17646000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43385ffce2e4669b21fc1be17646000" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33307 } } }
I20260927 15:20:36.406240 11915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:36.416594 11920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:36.427340 11920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:36.429605 11920 catalog_manager.cc:1259] Loaded cluster ID: 5f87b6d2647a417cbb229204578ed798
I20260927 15:20:36.429937 11920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:36.437070 11920 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:36.438896 11920 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d43385ffce2e4669b21fc1be17646000: Loaded TSK: 0
I20260927 15:20:36.440582 11920 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:20:36.710315 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:36.710602 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:36.710726 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:36.718257 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:36.718374 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:36.718616 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:36.728559 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.65.keytab
--principal=oryx/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:37517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=41859
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:36.730048 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:36.731323 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:36.747566 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.747730 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.748870 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:36.751765 11912 server_base.cc:1034] running on GCE node
I20260927 15:20:36.752511 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:36.753618 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:36.755121 11912 hybrid_clock.cc:648] HybridClock initialized: now 1790522436755024 us; error 97 us; skew 500 ppm
I20260927 15:20:36.755475 11912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:36 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522436, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:36.763797 11912 init.cc:377] Logged in from keytab as oryx/127.2.246.65@KRBTEST.COM (short username oryx)
I20260927 15:20:36.765667 11912 webserver.cc:460] Webserver started at http://127.2.246.65:41859/ using document root <none> and password file <none>
I20260927 15:20:36.766664 11912 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:36.766790 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:36.772698 11912 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:20:36.775900 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.777781 11912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:20:36.778057 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "d75760c8223546c2b6bf48a6f27e5758"
format_stamp: "Formatted at 2026-09-27 15:20:33 on dist-test-slave-14jq"
server_key: "9283801a9c3c8c5807058952a417bb0e"
server_key_iv: "f1bdaa1db1a54be5a87f60d529de6223"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.779075 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:36.801437 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:36.806595 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:36.806931 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:36.808094 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:36.809679 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:36.809809 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.809983 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:36.810081 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.857720 11912 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:37517
I20260927 15:20:36.857765 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:37517 every 8 connection(s)
I20260927 15:20:36.859226 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:36.862116 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 11912
I20260927 15:20:36.862757 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:38735
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:33307
--webserver_port=34243
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:20:36 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:36 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:20:37.297937 12063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:37.298251 12063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:37.298404 12063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:37.306083 12063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:37.306200 12063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:37.306367 12063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:37.316506 12063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.66.keytab
--principal=oryx/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:38735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=34243
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:37.318223 12063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:37.319876 12063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:37.343550 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.343948 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.346771 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:37.347024 12063 server_base.cc:1034] running on GCE node
I20260927 15:20:37.347831 12063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:37.348965 12063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:37.350198 12063 hybrid_clock.cc:648] HybridClock initialized: now 1790522437350155 us; error 75 us; skew 500 ppm
I20260927 15:20:37.350602 12063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:37 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:37.354646 12063 init.cc:377] Logged in from keytab as oryx/127.2.246.66@KRBTEST.COM (short username oryx)
I20260927 15:20:37.356782 12063 webserver.cc:460] Webserver started at http://127.2.246.66:34243/ using document root <none> and password file <none>
I20260927 15:20:37.357604 12063 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:37.357780 12063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:37.365029 12063 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:20:37.369588 12079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.372385 12063 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 15:20:37.372658 12063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "8306ebb409784e93ae07998b57c43448"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "d51f913288df4b698bfbac859e51df60"
server_key_iv: "a216bcf816486c0913fd84d34f63b860"
server_key_version: "encryptionkey@0"
I20260927 15:20:37.373771 12063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:37.403649 12063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:37.408836 12063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:37.409227 12063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:37.410665 12063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:37.412461 12063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:37.412614 12063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.412747 12063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:37.412839 12063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.464875 12063 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:38735
I20260927 15:20:37.464919 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:38735 every 8 connection(s)
I20260927 15:20:37.466681 12063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:37.467758 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12063
I20260927 15:20:37.468806 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:40407
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:33307
--webserver_port=37579
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:20:37 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:37 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:20:37.847431 12196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:37.847784 12196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:37.847955 12196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:37.856163 12196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:37.856329 12196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:37.856513 12196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:37.871224 12196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/krb5kdc/oryx_127.2.246.67.keytab
--principal=oryx/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:40407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=37579
--tserver_master_addrs=127.2.246.126:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:37.872769 12196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:37.874526 12196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:37.892859 12202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.892972 12203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.911623 12205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:37.912734 12196 server_base.cc:1034] running on GCE node
I20260927 15:20:37.913516 12196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:37.914713 12196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:37.915992 12196 hybrid_clock.cc:648] HybridClock initialized: now 1790522437915908 us; error 64 us; skew 500 ppm
I20260927 15:20:37.916378 12196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:20:37 dist-test-slave-14jq krb5kdc[11323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:37.919927 12196 init.cc:377] Logged in from keytab as oryx/127.2.246.67@KRBTEST.COM (short username oryx)
I20260927 15:20:37.923298 12196 webserver.cc:460] Webserver started at http://127.2.246.67:37579/ using document root <none> and password file <none>
I20260927 15:20:37.924098 12196 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:37.924266 12196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:37.930071 12196 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:20:37.932929 12212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.934577 12196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:37.934795 12196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "337be0d736a447fd90b7ff0308f6e176"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-14jq"
server_key: "b44443559717d03e4c4f6e348168c1e7"
server_key_iv: "24c389bbda98054c55253f0bd53b12d9"
server_key_version: "encryptionkey@0"
I20260927 15:20:37.935894 12196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:37.960529 12196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:37.964986 12196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:37.965258 12196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:37.966403 12196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:37.967960 12196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:37.968075 12196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.968205 12196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:37.968289 12196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.020408 12196 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:40407
I20260927 15:20:38.020483 12324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:40407 every 8 connection(s)
I20260927 15:20:38.023123 12196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:38.030668 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12196
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:38.080369 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.032525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.2.246.126:33307)
0927 15:20:38.047915 (+ 15390us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.047926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.051824 (+ 3898us) server_negotiation.cc:407] Connection header received
0927 15:20:38.051973 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.051985 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.052155 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.052343 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.057791 (+ 5448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.059286 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.061465 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.062226 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.062352 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.066985 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.066999 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.067037 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:38.067100 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.073385 (+ 6285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.074838 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.074854 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.074870 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.075169 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.075755 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.075764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.075771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.076633 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.077026 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.078039 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.078782 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15159,"thread_start_us":15422,"threads_started":1}
W20260927 15:20:38.452194 12062 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.574s user 0.006s sys 0.009s
I20260927 15:20:38.455315 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.863066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:60075)
0927 15:20:36.864401 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.864456 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.866569 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.868608 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.868618 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.869321 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:36.870142 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.870165 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.872372 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.872381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.874904 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.874929 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.876423 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.876432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.876833 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.877863 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:36.877912 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.453543 (+1575631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":765,"threads_started":1}
W20260927 15:20:38.455597 12061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:36.862865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:60075 (local address 127.2.246.126:33307)
0927 15:20:36.863928 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.863949 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.864650 (+ 701us) server_negotiation.cc:407] Connection header received
0927 15:20:36.867282 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.867308 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.867866 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.868386 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:36.870406 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.872128 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.875201 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.876058 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.876277 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.454936 (+1578659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:60075: BlockingRecv error: recv got EOF from 127.2.246.65:60075 (error 108)
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":407,"threads_started":1}
W20260927 15:20:38.457125 12059 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33307 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.246.126:33307: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:38.486016 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.458283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:57809)
0927 15:20:38.458730 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.458759 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.458930 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.459821 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.459829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.460543 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.460862 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.460880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.463296 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.463307 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.465976 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.466013 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.469344 (+ 3331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.469360 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.469959 (+ 599us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.471287 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.471341 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.485470 (+ 14129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242}
W20260927 15:20:38.486481 12061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:38.458521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:57809 (local address 127.2.246.126:33307)
0927 15:20:38.458854 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.458865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.458906 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:20:38.459200 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.459210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.459344 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.459464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.461144 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.462958 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.466378 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.467926 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.468287 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.486038 (+ 17751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:57809: BlockingRecv error: recv got EOF from 127.2.246.65:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:38.513809 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.487091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:58145)
0927 15:20:38.492989 (+ 5898us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.493021 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.493247 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.494182 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.494190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.494941 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.495277 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.495296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.497602 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.497611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.500184 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.500212 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.502949 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.502970 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.503339 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.505031 (+ 1692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.505075 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.513271 (+ 8196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5718,"thread_start_us":11391,"threads_started":1}
W20260927 15:20:38.514642 12061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:38.487691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:58145 (local address 127.2.246.126:33307)
0927 15:20:38.487930 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.487937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.493167 (+ 5230us) server_negotiation.cc:407] Connection header received
0927 15:20:38.493550 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.493561 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.493693 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.493800 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.495531 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.497283 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.500504 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.501977 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.502285 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.513994 (+ 11709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:58145: BlockingRecv error: recv got EOF from 127.2.246.65:58145 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 15:20:38.516327 12059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:20:38.942734 12194 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.432s user 0.006s sys 0.012s
W20260927 15:20:38.944895 12195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:37.474848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36055 (local address 127.2.246.126:33307)
0927 15:20:37.494408 (+ 19560us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:37.494418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:37.496635 (+ 2217us) server_negotiation.cc:407] Connection header received
0927 15:20:37.499488 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:37.499502 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:37.499693 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:37.499907 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:37.502945 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.505146 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.508094 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.509266 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.509512 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.944329 (+1434817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:36055: BlockingRecv error: recv got EOF from 127.2.246.66:36055 (error 108)
Metrics: {"server-negotiator.queue_time_us":19363,"thread_start_us":19234,"threads_started":1}
I20260927 15:20:38.945601 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:37.473621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:36055)
0927 15:20:37.495245 (+ 21624us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:37.495317 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:37.498831 (+ 3514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:37.500315 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:37.500335 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:37.501333 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:37.502538 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:37.502575 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.505441 (+ 2866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.505451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:37.507829 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:37.507847 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.509602 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.509610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:37.509974 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:37.511111 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:37.511144 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.943873 (+1432729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21096,"thread_start_us":20702,"threads_started":1}
W20260927 15:20:38.947058 12192 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33307 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.246.126:33307: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:38.960105 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.948124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:52653)
0927 15:20:38.948401 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.948432 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.948607 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.949022 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.949027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.949498 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.949705 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.949716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.950949 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.950956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.952069 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.952082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.952942 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.952949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.953171 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.954122 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.954177 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.959691 (+ 5514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 15:20:38.960232 12195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:38.948113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:52653 (local address 127.2.246.126:33307)
0927 15:20:38.948328 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.948335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.948508 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:20:38.948777 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.948789 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.948866 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.948988 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.949910 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.950765 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.952244 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.952722 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.952892 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.959945 (+ 7053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:52653: BlockingRecv error: recv got EOF from 127.2.246.66:52653 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:38 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:38.973104 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.961579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:59655)
0927 15:20:38.962096 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.962119 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.962316 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.962729 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.962733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.963162 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.963382 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.963392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.964534 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.964542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.965595 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.965609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.966495 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.966502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.966733 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.967504 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.967524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.972729 (+ 5205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381}
W20260927 15:20:38.973455 12195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:38.961822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:59655 (local address 127.2.246.126:33307)
0927 15:20:38.962058 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.962066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.962216 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:20:38.962473 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.962478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.962565 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.962673 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.963551 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.964376 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.965768 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.966304 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.966448 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.973023 (+ 6575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:59655: BlockingRecv error: recv got EOF from 127.2.246.66:59655 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:20:38.974069 12192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:20:39.337996 12327 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.269s user 0.000s sys 0.022s
W20260927 15:20:39.339468 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:38.032880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:50883 (local address 127.2.246.126:33307)
0927 15:20:38.050469 (+ 17589us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.050478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.054843 (+ 4365us) server_negotiation.cc:407] Connection header received
0927 15:20:38.057625 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.057632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.057758 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.057926 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.061210 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.062576 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.065683 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.066519 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.066646 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.339148 (+1272502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:50883: BlockingRecv error: recv got EOF from 127.2.246.67:50883 (error 108)
Metrics: {"server-negotiator.queue_time_us":17444,"thread_start_us":17310,"threads_started":1}
I20260927 15:20:39.340175 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.029321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:50883)
0927 15:20:38.053606 (+ 24285us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.053688 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.056911 (+ 3223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.058721 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.058741 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.059761 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.060880 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.060918 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.062825 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.062831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.065412 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.065438 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.067106 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.067114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.067560 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.068613 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.068670 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.338944 (+1270274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23815,"thread_start_us":23408,"threads_started":1}
W20260927 15:20:39.341221 12325 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33307 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.246.126:33307: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:39.353751 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.342229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:53049)
0927 15:20:39.342476 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.342497 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.342638 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.343002 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.343005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.343446 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.343661 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.343672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.345393 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.345399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.346692 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.346713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.347561 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.347568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.347752 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.348448 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:39.348470 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.353449 (+ 4979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 15:20:39.354329 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:39.342289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:53049 (local address 127.2.246.126:33307)
0927 15:20:39.342459 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.342463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.342577 (+ 114us) server_negotiation.cc:407] Connection header received
0927 15:20:39.342765 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.342769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.342840 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.342947 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.343855 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.345192 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.346883 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.347323 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.347484 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.354065 (+ 6581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:53049: BlockingRecv error: recv got EOF from 127.2.246.67:53049 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:39.366473 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.354754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:35641)
0927 15:20:39.354924 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.354947 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.355079 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.355704 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.355707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.356125 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.356322 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.356332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.357706 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.357711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.359337 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.359359 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.360647 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.360655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.360974 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.361756 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:39.361786 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.366230 (+ 4444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:20:39.366875 12325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:20:39.366976 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:39.354988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:35641 (local address 127.2.246.126:33307)
0927 15:20:39.355277 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.355282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.355304 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:39.355391 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.355395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.355472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.355592 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.356543 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.357562 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.359529 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.360374 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.360562 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.366656 (+ 6094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:35641: BlockingRecv error: recv got EOF from 127.2.246.67:35641 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:39.528066 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.517383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:56361)
0927 15:20:39.518099 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.518119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.518293 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.518823 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.518828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.519385 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.519595 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.519611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.520794 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.520801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.521783 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.521794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.522581 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.522588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.522763 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.523394 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:39.523413 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.527655 (+ 4242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":335,"threads_started":1}
W20260927 15:20:39.528167 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:39.517604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:56361 (local address 127.2.246.126:33307)
0927 15:20:39.517930 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.517970 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.518307 (+ 337us) server_negotiation.cc:407] Connection header received
0927 15:20:39.518417 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.518422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.518553 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.518715 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.519780 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.520642 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.521951 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.522367 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.522520 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.527912 (+ 5392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:56361: BlockingRecv error: recv got EOF from 127.2.246.65:56361 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:39 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:39.986016 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.974875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:41605)
0927 15:20:39.975549 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.975568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.975752 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.976124 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.976129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.976654 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.976825 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.976834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.977953 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.977960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.979055 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.979068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.979881 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.979887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.980074 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.980891 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:39.980912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.985676 (+ 4764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":345,"threads_started":1}
W20260927 15:20:39.986555 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:39.975011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:41605 (local address 127.2.246.126:33307)
0927 15:20:39.975262 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.975270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.975662 (+ 392us) server_negotiation.cc:407] Connection header received
0927 15:20:39.975888 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.975894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.975970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.976084 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.976967 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.977780 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.979202 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.979671 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.979818 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.986167 (+ 6349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:41605: BlockingRecv error: recv got EOF from 127.2.246.66:41605 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:20:40.383841 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:40.367849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:32829 (local address 127.2.246.126:33307)
0927 15:20:40.368121 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:40.368132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:40.368768 (+ 636us) server_negotiation.cc:407] Connection header received
0927 15:20:40.369004 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:40.369014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:40.369143 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:40.369338 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:40.371650 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.372493 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.375149 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.375711 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.375829 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.383614 (+ 7785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:32829: BlockingRecv error: recv got EOF from 127.2.246.67:32829 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:20:40.384363 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:40.367759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:32829)
0927 15:20:40.368622 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:40.368643 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:40.368812 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:40.369641 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:40.369650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:40.371046 (+ 1396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:40.371375 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.371397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.372667 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.372676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.374862 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.374892 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.375970 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.375980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.376237 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.377058 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:40.377091 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:40.384125 (+ 7034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":591,"threads_started":1}
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:40.541055 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:40.529465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:44765)
0927 15:20:40.530243 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:40.530271 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:40.530443 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:40.530916 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:40.530921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:40.531446 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:40.531699 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.531710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.532960 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.532965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.534315 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.534340 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.535233 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.535240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.535489 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.536252 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:40.536284 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:40.540734 (+ 4450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":506,"threads_started":1}
W20260927 15:20:40.541314 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:40.529597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:44765 (local address 127.2.246.126:33307)
0927 15:20:40.529768 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:40.529772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:40.530381 (+ 609us) server_negotiation.cc:407] Connection header received
0927 15:20:40.530609 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:40.530613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:40.530686 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:40.530805 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:40.531884 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.532736 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.534546 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.534991 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.535108 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.540949 (+ 5841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:44765: BlockingRecv error: recv got EOF from 127.2.246.65:44765 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:40 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:40.998435 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:40.987224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:34075)
0927 15:20:40.987814 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:40.987861 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:40.988038 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:40.988374 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:40.988378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:40.988896 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:40.989220 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.989232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.990448 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.990453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.991585 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:40.991600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.992534 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.992541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:40.992754 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.993685 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:40.993721 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:40.998178 (+ 4457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":339,"threads_started":1}
W20260927 15:20:40.998868 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:40.987387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:34075 (local address 127.2.246.126:33307)
0927 15:20:40.987710 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:40.987716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:40.987973 (+ 257us) server_negotiation.cc:407] Connection header received
0927 15:20:40.988153 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:40.988157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:40.988222 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:40.988314 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:40.989366 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.990300 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.991786 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:40.992300 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:40.992411 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:40.998542 (+ 6131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:34075: BlockingRecv error: recv got EOF from 127.2.246.66:34075 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 15:20:41 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:41 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:41.397086 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:41.385564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:58569)
0927 15:20:41.386157 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:41.386182 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:41.386319 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:41.386979 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:41.386984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:41.387589 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:41.387856 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:41.387870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.389257 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.389264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:41.390494 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:41.390512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.391329 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.391345 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:41.391549 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.392320 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:41.392346 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:41.396795 (+ 4449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":345,"threads_started":1}
W20260927 15:20:41.397550 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:41.385960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:58569 (local address 127.2.246.126:33307)
0927 15:20:41.386187 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:41.386193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:41.386258 (+ 65us) server_negotiation.cc:407] Connection header received
0927 15:20:41.386601 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:41.386608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:41.386715 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:41.386883 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:41.388051 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.389056 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.390662 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.391140 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.391296 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.397212 (+ 5916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:58569: BlockingRecv error: recv got EOF from 127.2.246.67:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:20:41 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:41 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:41.560355 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:41.542449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:50577)
0927 15:20:41.543121 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:41.543141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:41.543311 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:41.543860 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:41.543864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:41.544367 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:41.544605 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:41.544615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.546332 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.546339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:41.547789 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:41.547810 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.548671 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.548675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:41.548819 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.554529 (+ 5710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:41.554568 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:41.559824 (+ 5256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":365,"threads_started":1}
W20260927 15:20:41.560529 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:41.542500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:50577 (local address 127.2.246.126:33307)
0927 15:20:41.542773 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:41.542778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:41.543268 (+ 490us) server_negotiation.cc:407] Connection header received
0927 15:20:41.543532 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:41.543537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:41.543623 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:41.543757 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:41.544869 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.546090 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.547970 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.548438 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.548630 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.560135 (+ 11505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:50577: BlockingRecv error: recv got EOF from 127.2.246.65:50577 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:42.010798 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:41.999899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:36829)
0927 15:20:42.000468 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:42.000487 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:42.000627 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:42.001150 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:42.001154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:42.001575 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:42.001783 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.001794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.003322 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.003326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.004431 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.004443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.005289 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.005295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.005419 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.006206 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:42.006232 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:42.010559 (+ 4327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":285,"threads_started":1}
W20260927 15:20:42.011251 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:42.000077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36829 (local address 127.2.246.126:33307)
0927 15:20:42.000249 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:42.000254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:42.000651 (+ 397us) server_negotiation.cc:407] Connection header received
0927 15:20:42.000752 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:42.000756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:42.000935 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:42.001080 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:42.002104 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.003103 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.004603 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.005123 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.005260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.010893 (+ 5633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:36829: BlockingRecv error: recv got EOF from 127.2.246.66:36829 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:42.408818 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:42.398553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:46141)
0927 15:20:42.399224 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:42.399243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:42.399431 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:42.399841 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:42.399846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:42.400285 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:42.400445 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.400456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.401530 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.401535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.402611 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.402623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.403547 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.403553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.403728 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.404395 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:42.404412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:42.408526 (+ 4114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":527,"threads_started":1}
W20260927 15:20:42.409271 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:42.398887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:46141 (local address 127.2.246.126:33307)
0927 15:20:42.399086 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:42.399091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:42.399358 (+ 267us) server_negotiation.cc:407] Connection header received
0927 15:20:42.399608 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:42.399621 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:42.399696 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:42.399816 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:42.400599 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.401350 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.402772 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.403359 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.403529 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.408833 (+ 5304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:46141: BlockingRecv error: recv got EOF from 127.2.246.67:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:42 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:42.573678 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:42.561717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:53127)
0927 15:20:42.562791 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:42.562817 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:42.563006 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:42.563400 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:42.563404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:42.563909 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:42.564214 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.564226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.565293 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.565298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.566449 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:42.566464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.567263 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.567268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:42.567421 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.568233 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:42.568256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:42.573167 (+ 4911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":747,"threads_started":1}
W20260927 15:20:42.573890 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:42.561814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:53127 (local address 127.2.246.126:33307)
0927 15:20:42.562077 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:42.562083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:42.562973 (+ 890us) server_negotiation.cc:407] Connection header received
0927 15:20:42.563124 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:42.563128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:42.563195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:42.563299 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:42.564388 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.565137 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.566622 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.567068 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.567181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.573466 (+ 6285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:53127: BlockingRecv error: recv got EOF from 127.2.246.65:53127 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:43.022382 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:43.012241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:38413)
0927 15:20:43.012823 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:43.012841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:43.012976 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:43.013438 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:43.013441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:43.013920 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:43.014102 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.014112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.015426 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.015430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.016451 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.016463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.017576 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.017580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.017708 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.018456 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:43.018473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:43.022011 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
W20260927 15:20:43.022570 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:43.012350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:38413 (local address 127.2.246.126:33307)
0927 15:20:43.012584 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:43.012589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:43.012929 (+ 340us) server_negotiation.cc:407] Connection header received
0927 15:20:43.013133 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:43.013137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:43.013217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:43.013337 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:43.014299 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.015255 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.016735 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.017367 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.017509 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.022287 (+ 4778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:38413: BlockingRecv error: recv got EOF from 127.2.246.66:38413 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:43.421798 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:43.410326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:56399)
0927 15:20:43.410998 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:43.411018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:43.411176 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:43.411711 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:43.411715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:43.412221 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:43.412476 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.412488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.413764 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.413769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.415030 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.415047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.415945 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.415950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.416112 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.417169 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:43.417197 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:43.421415 (+ 4218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":356,"threads_started":1}
W20260927 15:20:43.422000 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:43.410418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:56399 (local address 127.2.246.126:33307)
0927 15:20:43.410689 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:43.410694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:43.411118 (+ 424us) server_negotiation.cc:407] Connection header received
0927 15:20:43.411379 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:43.411383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:43.411458 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:43.411629 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:43.412681 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.413505 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.415221 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.415666 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.415879 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.421705 (+ 5826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:56399: BlockingRecv error: recv got EOF from 127.2.246.67:56399 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:43 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:43.584777 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:43.574868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:40759)
0927 15:20:43.575398 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:43.575415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:43.575574 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:43.576091 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:43.576095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:43.576510 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:43.576684 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.576692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.577919 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.577924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.579049 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:43.579061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.579978 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.579982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:43.580114 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.580806 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:43.580825 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:43.584488 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
W20260927 15:20:43.585268 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:43.574987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:40759 (local address 127.2.246.126:33307)
0927 15:20:43.575397 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:43.575402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:43.575513 (+ 111us) server_negotiation.cc:407] Connection header received
0927 15:20:43.575777 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:43.575782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:43.575882 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:43.576015 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:43.576878 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.577704 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.579226 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:43.579703 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:43.579961 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:43.584865 (+ 4904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:40759: BlockingRecv error: recv got EOF from 127.2.246.65:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:44.033737 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:44.023898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:36309)
0927 15:20:44.024545 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:44.024567 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:44.024728 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:44.025192 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:44.025199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:44.025674 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:44.025929 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.025940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.027092 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.027096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.028153 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.028165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.028899 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.028903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.029034 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.029736 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:44.029755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:44.033463 (+ 3708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":295,"threads_started":1}
W20260927 15:20:44.034291 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:44.024179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36309 (local address 127.2.246.126:33307)
0927 15:20:44.024397 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:44.024403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:44.024694 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:20:44.024903 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:44.024909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:44.025007 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:44.025129 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:44.026068 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.026917 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.028295 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.028754 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.028856 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.033928 (+ 5072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:36309: BlockingRecv error: recv got EOF from 127.2.246.66:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:44.435932 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:44.423485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:60277)
0927 15:20:44.424227 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:44.424248 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:44.424425 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:44.425017 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:44.425021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:44.425526 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:44.426040 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.426065 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.427388 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.427394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.428464 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.428481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.429446 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.429452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.429706 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.430868 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:44.430903 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:44.435603 (+ 4700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":425,"threads_started":1}
W20260927 15:20:44.436299 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:44.423641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:60277 (local address 127.2.246.126:33307)
0927 15:20:44.423865 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:44.423870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:44.424412 (+ 542us) server_negotiation.cc:407] Connection header received
0927 15:20:44.424688 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:44.424693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:44.424774 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:44.424991 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:44.426272 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.427170 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.428696 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.429169 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.429388 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.435896 (+ 6508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:60277: BlockingRecv error: recv got EOF from 127.2.246.67:60277 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:44 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:44.598093 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:44.586028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:46411)
0927 15:20:44.586718 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:44.586735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:44.587038 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:44.587737 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:44.587744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:44.588307 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:44.588529 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.588541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.589794 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.589802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.590921 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:44.590936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.592096 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.592102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:44.592314 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.593066 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:44.593087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:44.597637 (+ 4550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":402,"threads_started":1}
W20260927 15:20:44.598278 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:44.586222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:46411 (local address 127.2.246.126:33307)
0927 15:20:44.586585 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:44.586591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:44.586857 (+ 266us) server_negotiation.cc:407] Connection header received
0927 15:20:44.587370 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:44.587376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:44.587487 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:44.587627 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:44.588744 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.589612 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.591106 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.591886 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.592071 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.597979 (+ 5908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:46411: BlockingRecv error: recv got EOF from 127.2.246.65:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:45.046005 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:45.035303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:55625)
0927 15:20:45.036146 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:45.036173 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:45.036359 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:45.036757 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:45.036762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:45.037232 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:45.037384 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.037394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.038582 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.038589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.039632 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.039643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.040406 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.040413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.040604 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.041342 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:45.041361 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:45.045624 (+ 4263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":554,"threads_started":1}
W20260927 15:20:45.046314 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:45.035527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:55625 (local address 127.2.246.126:33307)
0927 15:20:45.035887 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:45.035894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:45.036280 (+ 386us) server_negotiation.cc:407] Connection header received
0927 15:20:45.036504 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:45.036515 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:45.036596 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:45.036727 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:45.037544 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.038418 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.039785 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.040227 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.040373 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.045991 (+ 5618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:55625: BlockingRecv error: recv got EOF from 127.2.246.66:55625 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:45.449646 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:45.437207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:40255)
0927 15:20:45.437730 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:45.437748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:45.437908 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:45.438402 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:45.438408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:45.439004 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:45.439326 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.439343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.440781 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.440786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.441910 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.441926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.442892 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.442898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.443067 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.443884 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:45.443905 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:45.449333 (+ 5428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":317,"threads_started":1}
W20260927 15:20:45.449947 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:45.437305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:40255 (local address 127.2.246.126:33307)
0927 15:20:45.437588 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:45.437597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:45.437865 (+ 268us) server_negotiation.cc:407] Connection header received
0927 15:20:45.438040 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:45.438044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:45.438120 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:45.438220 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:45.439560 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.440542 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.442133 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.442651 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.442786 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.449641 (+ 6855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:40255: BlockingRecv error: recv got EOF from 127.2.246.67:40255 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:45 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:45.610739 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:45.599547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:55909)
0927 15:20:45.600308 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:45.600328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:45.600488 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:45.601033 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:45.601037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:45.601583 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:45.601938 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.601950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.603330 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.603336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.604492 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:45.604534 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.605388 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.605392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:45.605508 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.606340 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:45.606361 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:45.610455 (+ 4094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":354,"threads_started":1}
W20260927 15:20:45.611085 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:45.599724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:55909 (local address 127.2.246.126:33307)
0927 15:20:45.599917 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:45.599921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:45.600453 (+ 532us) server_negotiation.cc:407] Connection header received
0927 15:20:45.600758 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:45.600767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:45.600859 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:45.600978 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:45.602106 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.603065 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.604730 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:45.605218 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:45.605336 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:45.610789 (+ 5453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:55909: BlockingRecv error: recv got EOF from 127.2.246.65:55909 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:46.058565 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:46.047436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:45657)
0927 15:20:46.048436 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:46.048469 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:46.048646 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:46.049040 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:46.049044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:46.049534 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:46.049737 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.049746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.051045 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.051049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.052116 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.052135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.053082 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.053087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.053218 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.054123 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:46.054141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:46.058262 (+ 4121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":606,"threads_started":1}
W20260927 15:20:46.059193 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:46.047545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:45657 (local address 127.2.246.126:33307)
0927 15:20:46.047914 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:46.047924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:46.048603 (+ 679us) server_negotiation.cc:407] Connection header received
0927 15:20:46.048768 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:46.048771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:46.048851 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:46.048951 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:46.050005 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.050864 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.052322 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.052885 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.053027 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.058815 (+ 5788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:45657: BlockingRecv error: recv got EOF from 127.2.246.66:45657 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:46.461737 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:46.451217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:43053)
0927 15:20:46.451934 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:46.451954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:46.452182 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:46.452643 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:46.452651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:46.453117 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:46.453305 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.453314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.454526 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.454539 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.455775 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.455788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.456589 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.456595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.456773 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.457527 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:46.457546 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:46.461424 (+ 3878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":337,"threads_started":1}
W20260927 15:20:46.462173 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:46.451337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:43053 (local address 127.2.246.126:33307)
0927 15:20:46.451615 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:46.451623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:46.452096 (+ 473us) server_negotiation.cc:407] Connection header received
0927 15:20:46.452335 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:46.452348 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:46.452417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:46.452536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:46.453469 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.454364 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.455934 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.456406 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.456567 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.461820 (+ 5253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:43053: BlockingRecv error: recv got EOF from 127.2.246.67:43053 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:46 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:46.623955 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:46.612330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:54927)
0927 15:20:46.613037 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:46.613056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:46.613208 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:46.613756 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:46.613762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:46.614314 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:46.614529 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.614539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.615892 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.615897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.617468 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:46.617488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.618618 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.618624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:46.618781 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.619593 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:46.619614 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:46.623661 (+ 4047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":378,"threads_started":1}
W20260927 15:20:46.624218 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:46.612638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:54927 (local address 127.2.246.126:33307)
0927 15:20:46.612852 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:46.612857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:46.613158 (+ 301us) server_negotiation.cc:407] Connection header received
0927 15:20:46.613384 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:46.613389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:46.613525 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:46.613659 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:46.614736 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.615665 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.617671 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.618382 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.618505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.623904 (+ 5399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:54927: BlockingRecv error: recv got EOF from 127.2.246.65:54927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:47.076877 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:47.059805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:34071)
0927 15:20:47.060412 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:47.060429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:47.060612 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:47.061018 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:47.061022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:47.061422 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:47.061640 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.061650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.063065 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.063069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.064102 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.064116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.064954 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.064958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.065084 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.072012 (+ 6928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:47.072033 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:47.076604 (+ 4571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":316,"threads_started":1}
W20260927 15:20:47.077250 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:47.060032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:34071 (local address 127.2.246.126:33307)
0927 15:20:47.060276 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:47.060282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:47.060557 (+ 275us) server_negotiation.cc:407] Connection header received
0927 15:20:47.060764 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:47.060769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:47.060847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:47.060969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:47.061882 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.062876 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.064276 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.064756 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.064876 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.076911 (+ 12035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:34071: BlockingRecv error: recv got EOF from 127.2.246.66:34071 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:47.476169 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:47.463055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:38907)
0927 15:20:47.463804 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:47.463834 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:47.464077 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:47.464601 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:47.464606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:47.465150 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:47.465380 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.465392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.467418 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.467424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.468952 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.468966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.470094 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.470101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.470289 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.471101 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:47.471125 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:47.475852 (+ 4727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":342,"threads_started":1}
W20260927 15:20:47.476455 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:47.463270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38907 (local address 127.2.246.126:33307)
0927 15:20:47.463521 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:47.463528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:47.463995 (+ 467us) server_negotiation.cc:407] Connection header received
0927 15:20:47.464274 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:47.464279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:47.464364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:47.464535 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:47.465899 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.467154 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.469123 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.469759 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.470011 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.476168 (+ 6157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:38907: BlockingRecv error: recv got EOF from 127.2.246.67:38907 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:47 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:47.636228 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:47.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:38543)
0927 15:20:47.625978 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:47.626001 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:47.626235 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:47.626875 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:47.626879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:47.627388 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:47.627576 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.627584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.628715 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.628721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.629871 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:47.629884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.630694 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.630699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:47.630863 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.631667 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:47.631688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:47.635945 (+ 4257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":459,"threads_started":1}
W20260927 15:20:47.636479 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:47.625185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:38543 (local address 127.2.246.126:33307)
0927 15:20:47.625440 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:47.625449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:47.626137 (+ 688us) server_negotiation.cc:407] Connection header received
0927 15:20:47.626495 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:47.626503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:47.626631 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:47.626753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:47.627759 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.628537 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.630041 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:47.630499 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:47.630619 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:47.636219 (+ 5600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:38543: BlockingRecv error: recv got EOF from 127.2.246.65:38543 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:48.090185 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:48.078361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:35725)
0927 15:20:48.079070 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:48.079090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:48.079322 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:48.079818 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:48.079822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:48.080471 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:48.080716 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.080729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.082032 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.082037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.083398 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.083413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.084430 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.084436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.084625 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.085464 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:48.085484 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:48.089801 (+ 4317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
W20260927 15:20:48.090344 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:48.078492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:35725 (local address 127.2.246.126:33307)
0927 15:20:48.078848 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:48.078857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:48.079257 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:20:48.079524 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:48.079530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:48.079625 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:48.079762 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:48.080905 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.081860 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.083613 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.084208 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.084339 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.090093 (+ 5754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:35725: BlockingRecv error: recv got EOF from 127.2.246.66:35725 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:48.489001 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:48.477640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:49121)
0927 15:20:48.478361 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:48.478384 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:48.478567 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:48.479173 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:48.479177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:48.479853 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:48.480005 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.480013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.481129 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.481138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.482346 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.482358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.483313 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.483317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.483443 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.484269 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:48.484287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:48.488560 (+ 4273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
W20260927 15:20:48.489125 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:48.477790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:49121 (local address 127.2.246.126:33307)
0927 15:20:48.478113 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:48.478121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:48.478557 (+ 436us) server_negotiation.cc:407] Connection header received
0927 15:20:48.478878 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:48.478885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:48.478982 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:48.479118 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:48.480164 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.480953 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.482512 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.483150 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.483275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.488864 (+ 5589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:49121: BlockingRecv error: recv got EOF from 127.2.246.67:49121 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:48 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:48.647506 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:48.637407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:51865)
0927 15:20:48.638051 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:48.638070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:48.638257 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:48.638631 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:48.638635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:48.639119 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:48.639336 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.639350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.640517 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.640523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.641586 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:48.641597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.642438 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.642444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:48.642629 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.643333 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:48.643352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:48.647221 (+ 3869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":322,"threads_started":1}
W20260927 15:20:48.648031 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:48.637570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:51865 (local address 127.2.246.126:33307)
0927 15:20:48.637822 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:48.637828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:48.638163 (+ 335us) server_negotiation.cc:407] Connection header received
0927 15:20:48.638390 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:48.638396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:48.638491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:48.638599 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:48.639509 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.640360 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.641730 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.642270 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.642417 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.647713 (+ 5296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:51865: BlockingRecv error: recv got EOF from 127.2.246.65:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:49.103451 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:49.091605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:49431)
0927 15:20:49.092280 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:49.092301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:49.092466 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:49.092928 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:49.092934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:49.093529 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:49.093708 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.093718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.094868 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.094874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.096088 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.096103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.096883 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.096888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.097026 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.097783 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:49.097806 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:49.103150 (+ 5344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":394,"threads_started":1}
W20260927 15:20:49.104049 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:49.091821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:49431 (local address 127.2.246.126:33307)
0927 15:20:49.092030 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:49.092037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:49.092432 (+ 395us) server_negotiation.cc:407] Connection header received
0927 15:20:49.092625 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:49.092630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:49.092726 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:49.092845 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:49.093921 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.094687 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.096274 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.096673 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.096776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.103647 (+ 6871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:49431: BlockingRecv error: recv got EOF from 127.2.246.66:49431 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:49.500535 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:49.490327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:39893)
0927 15:20:49.491081 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:49.491099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:49.491290 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:49.491643 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:49.491649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:49.492144 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:49.492323 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.492333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.493476 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.493488 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.494550 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.494561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.495296 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.495302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.495482 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.496219 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:49.496238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:49.500138 (+ 3900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":447,"threads_started":1}
W20260927 15:20:49.500689 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:49.490543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:39893 (local address 127.2.246.126:33307)
0927 15:20:49.490843 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:49.490851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:49.491202 (+ 351us) server_negotiation.cc:407] Connection header received
0927 15:20:49.491440 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:49.491444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:49.491508 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:49.491615 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:49.492488 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.493311 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.494692 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.495136 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.495275 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.500425 (+ 5150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:39893: BlockingRecv error: recv got EOF from 127.2.246.67:39893 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:49 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:49.659178 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:49.648744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:58359)
0927 15:20:49.649390 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:49.649409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:49.649604 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:49.650147 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:49.650150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:49.650660 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:49.650850 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.650859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.651978 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.651983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.652947 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:49.652958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.653735 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.653739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:49.653928 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.654655 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:49.654673 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:49.658764 (+ 4091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":369,"threads_started":1}
W20260927 15:20:49.659350 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:49.648830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:58359 (local address 127.2.246.126:33307)
0927 15:20:49.649158 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:49.649165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:49.649510 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:20:49.649788 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:49.649793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:49.649959 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:49.650063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:49.651014 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.651827 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.653094 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:49.653580 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:49.653689 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:49.659049 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:58359: BlockingRecv error: recv got EOF from 127.2.246.65:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:50.116564 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:50.104926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:46523)
0927 15:20:50.105562 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:50.105581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:50.105730 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:50.106148 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:50.106151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:50.106622 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:50.106830 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.106839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.108053 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.108058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.109115 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.109130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.110199 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.110206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.110398 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.111283 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:50.111328 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:50.116209 (+ 4881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":333,"threads_started":1}
W20260927 15:20:50.116729 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:50.105107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:46523 (local address 127.2.246.126:33307)
0927 15:20:50.105356 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:50.105361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:50.105688 (+ 327us) server_negotiation.cc:407] Connection header received
0927 15:20:50.105883 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:50.105887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:50.105973 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:50.106107 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:50.107023 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.107878 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.109288 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.109930 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.110157 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.116447 (+ 6290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:46523: BlockingRecv error: recv got EOF from 127.2.246.66:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:50.514107 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:50.501964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:59075)
0927 15:20:50.502886 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:50.502908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:50.503126 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:50.503754 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:50.503760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:50.504373 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:50.504597 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.504607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.505768 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.505775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.506818 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.506833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.507661 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.507667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.507880 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.508770 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:50.508798 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:50.513569 (+ 4771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":508,"threads_started":1}
W20260927 15:20:50.514240 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:50.502209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:59075 (local address 127.2.246.126:33307)
0927 15:20:50.502670 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:50.502679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:50.503021 (+ 342us) server_negotiation.cc:407] Connection header received
0927 15:20:50.503288 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:50.503293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:50.503364 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:50.503722 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:50.504785 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.505597 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.506991 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.507452 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.507610 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.513977 (+ 6367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:59075: BlockingRecv error: recv got EOF from 127.2.246.67:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:50 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:50.670739 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:50.660301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:55123)
0927 15:20:50.660947 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:50.660965 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:50.661109 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:50.661440 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:50.661443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:50.661951 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:50.662146 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.662154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.663298 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.663302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.664297 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:50.664309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.665136 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.665140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:50.665290 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.666067 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:50.666086 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:50.670466 (+ 4380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":364,"threads_started":1}
W20260927 15:20:50.671061 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:50.660373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:55123 (local address 127.2.246.126:33307)
0927 15:20:50.660606 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:50.660611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:50.661057 (+ 446us) server_negotiation.cc:407] Connection header received
0927 15:20:50.661222 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:50.661225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:50.661293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:50.661399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:50.662312 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.663149 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.664460 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.664939 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.665089 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.670738 (+ 5649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:55123: BlockingRecv error: recv got EOF from 127.2.246.65:55123 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:51.129086 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:51.117990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:40595)
0927 15:20:51.118582 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:51.118602 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:51.118785 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:51.119189 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:51.119194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:51.119652 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:51.119848 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.119857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.121021 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.121026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.122366 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.122378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.123429 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.123435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.123892 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.124747 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:51.124769 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:51.128771 (+ 4002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":320,"threads_started":1}
W20260927 15:20:51.129310 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:51.118029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40595 (local address 127.2.246.126:33307)
0927 15:20:51.118340 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:51.118351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:51.118712 (+ 361us) server_negotiation.cc:407] Connection header received
0927 15:20:51.118918 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:51.118922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:51.118996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:51.119098 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:51.120024 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.120850 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.122537 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.123198 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.123356 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.129038 (+ 5682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:40595: BlockingRecv error: recv got EOF from 127.2.246.66:40595 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:51.526391 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:51.515533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:38255)
0927 15:20:51.516127 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:51.516173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:51.516379 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:51.516879 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:51.516884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:51.517321 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:51.517502 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.517512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.518936 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.518940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.519893 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.519904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.520859 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.520864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.520996 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.521698 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:51.521745 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:51.526081 (+ 4336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":331,"threads_started":1}
W20260927 15:20:51.526647 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:51.515756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38255 (local address 127.2.246.126:33307)
0927 15:20:51.516013 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:51.516019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:51.516340 (+ 321us) server_negotiation.cc:407] Connection header received
0927 15:20:51.516612 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:51.516619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:51.516723 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:51.516833 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:51.517655 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.518801 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.520067 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.520628 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.520777 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.526341 (+ 5564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:38255: BlockingRecv error: recv got EOF from 127.2.246.67:38255 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:51 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:51.683209 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:51.672266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:53861)
0927 15:20:51.672942 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:51.672967 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:51.673151 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:51.673657 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:51.673663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:51.674286 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:51.674483 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.674492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.675639 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.675644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.676707 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:51.676720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.677697 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.677703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:51.677914 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.678778 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:51.678798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:51.682797 (+ 3999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":385,"threads_started":1}
W20260927 15:20:51.683415 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:51.672443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:53861 (local address 127.2.246.126:33307)
0927 15:20:51.672734 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:51.672741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:51.673093 (+ 352us) server_negotiation.cc:407] Connection header received
0927 15:20:51.673314 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:51.673320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:51.673446 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:51.673585 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:51.674661 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.675443 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.676911 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:51.677445 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:51.677608 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:51.683171 (+ 5563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:53861: BlockingRecv error: recv got EOF from 127.2.246.65:53861 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.66@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:52.149959 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:52.130481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.66:39445)
0927 15:20:52.131059 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:52.131083 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:52.131257 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:52.131736 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:52.131740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:52.132300 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:52.132465 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.132475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.133998 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.134003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.135131 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.135144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.135899 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.135905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.136088 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.145017 (+ 8929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:52.145043 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:52.149573 (+ 4530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":362,"threads_started":1}
W20260927 15:20:52.150231 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:52.130763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:39445 (local address 127.2.246.126:33307)
0927 15:20:52.130980 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:52.130986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:52.131277 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:20:52.131389 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:52.131397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:52.131510 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:52.131655 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:52.132673 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.133760 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.135294 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.135737 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.135883 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.149909 (+ 14026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.66:39445: BlockingRecv error: recv got EOF from 127.2.246.66:39445 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.67@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:52.538467 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:52.527541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.67:33903)
0927 15:20:52.528147 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:52.528169 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:52.528334 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:52.528739 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:52.528742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:52.529225 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:52.529397 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.529406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.530669 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.530674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.531802 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.531816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.532581 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.532586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.532758 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.533501 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:52.533520 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:52.538199 (+ 4679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":329,"threads_started":1}
W20260927 15:20:52.538822 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:52.527710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:33903 (local address 127.2.246.126:33307)
0927 15:20:52.527971 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:52.527977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:52.528284 (+ 307us) server_negotiation.cc:407] Connection header received
0927 15:20:52.528464 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:52.528468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:52.528593 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:52.528693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:52.529588 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.530498 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.531981 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.532390 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.532552 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.538490 (+ 5938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.67:33903: BlockingRecv error: recv got EOF from 127.2.246.67:33903 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:20:52 dist-test-slave-14jq krb5kdc[11323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.246.65@KRBTEST.COM for oryx/127.2.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:20:52.695400 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:52.684588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33307 (local address 127.2.246.65:36205)
0927 15:20:52.685247 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:52.685272 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:52.685413 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:52.685940 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:52.685945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:52.686424 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:52.686630 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.686639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.687846 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.687851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.688860 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:52.688871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.689688 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.689692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:52.689822 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.690511 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:52.690530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:52.695114 (+ 4584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.246.126:33307: Server oryx/127.2.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":370,"threads_started":1}
W20260927 15:20:52.695739 12330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:52.684775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:36205 (local address 127.2.246.126:33307)
0927 15:20:52.685073 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:52.685078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:52.685392 (+ 314us) server_negotiation.cc:407] Connection header received
0927 15:20:52.685597 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:52.685601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:52.685666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:52.685771 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:52.686809 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.687659 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.689020 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.689495 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.689659 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.695376 (+ 5717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.246.65:36205: BlockingRecv error: recv got EOF from 127.2.246.65:36205 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260927 15:20:53.034412 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11912
I20260927 15:20:53.050004 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:53.036232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.2.246.65:37517)
0927 15:20:53.037166 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:53.037174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:53.037207 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:20:53.037273 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:53.037281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:53.037529 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:53.037787 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:53.039485 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.040827 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.042206 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.042858 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.043023 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:53.046001 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:53.046013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:53.046031 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:53.046084 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:53.049094 (+ 3010us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:20:53.049564 (+ 470us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39284: Request ticket server kudu/127.2.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":567,"threads_started":1}
W20260927 15:20:53.050146 12374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39284: Request ticket server kudu/127.2.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:20:53.050239 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11912: Not authorized: Client connection negotiation failed: client connection to 127.2.246.65:37517: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:20:53.050544 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11912
I20260927 15:20:53.071023 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12063
I20260927 15:20:53.088083 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:53.072559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.2.246.66:38735)
0927 15:20:53.073276 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:53.073285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:53.073316 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:20:53.073393 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:53.073401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:53.073726 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:53.074195 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:53.076223 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.077362 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.079095 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.079941 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.080079 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:53.083360 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:53.083373 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:53.083507 (+ 134us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:53.083604 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:53.087300 (+ 3696us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:20:53.087746 (+ 446us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35568: Request ticket server kudu/127.2.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":362,"threads_started":1}
W20260927 15:20:53.088250 12375 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35568: Request ticket server kudu/127.2.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:20:53.088385 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12063: Not authorized: Client connection negotiation failed: client connection to 127.2.246.66:38735: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:20:53.088588 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12063
I20260927 15:20:53.107921 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12196
I20260927 15:20:53.124711 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:53.109311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.2.246.67:40407)
0927 15:20:53.110167 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:53.110180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:53.110225 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:20:53.110334 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:53.110345 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:53.110757 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:53.111051 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:53.112420 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.114548 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.116017 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.116821 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.116985 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:53.120328 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:53.120340 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:53.120360 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:53.120423 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:53.123773 (+ 3350us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:20:53.124261 (+ 488us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38886: Request ticket server kudu/127.2.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":382,"threads_started":1}
W20260927 15:20:53.124859 12376 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38886: Request ticket server kudu/127.2.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:20:53.124869 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12196: Not authorized: Client connection negotiation failed: client connection to 127.2.246.67:40407: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:20:53.125237 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12196
I20260927 15:20:53.144677 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 11843
I20260927 15:20:53.146413 11905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:53.264722 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 11843
2026-09-27T15:20:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22314 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:53Z Disabled control of system clock
I20260927 15:20:53.350052 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33565
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:20:53.684764 12381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:53.685078 12381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:53.685196 12381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:53.692787 12381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:53.692907 12381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:53.692946 12381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:53.693034 12381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:53.693106 12381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:53.702766 12381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:53.704187 12381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:53.706039 12381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:53.719556 12386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:53.719424 12387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:53.719310 12389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:53.720845 12381 server_base.cc:1034] running on GCE node
I20260927 15:20:53.722007 12381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:53.723910 12381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:53.725179 12381 hybrid_clock.cc:648] HybridClock initialized: now 1790522453725125 us; error 46 us; skew 500 ppm
I20260927 15:20:53.725641 12381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:53.728549 12381 webserver.cc:460] Webserver started at http://127.2.246.126:44771/ using document root <none> and password file <none>
I20260927 15:20:53.729470 12381 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:53.729624 12381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:53.730104 12381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:53.733793 12381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "4facc2bf2b2940b4bec6dae9205a3fec"
format_stamp: "Formatted at 2026-09-27 15:20:53 on dist-test-slave-14jq"
server_key: "9ac6121b9957e6984e5690618e94771c"
server_key_iv: "986ea73cc44c36574ded5148486aa0f0"
server_key_version: "encryptionkey@0"
I20260927 15:20:53.734839 12381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "4facc2bf2b2940b4bec6dae9205a3fec"
format_stamp: "Formatted at 2026-09-27 15:20:53 on dist-test-slave-14jq"
server_key: "9ac6121b9957e6984e5690618e94771c"
server_key_iv: "986ea73cc44c36574ded5148486aa0f0"
server_key_version: "encryptionkey@0"
I20260927 15:20:53.742394 12381 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:20:53.747547 12395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:53.749527 12381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260927 15:20:53.749821 12381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "4facc2bf2b2940b4bec6dae9205a3fec"
format_stamp: "Formatted at 2026-09-27 15:20:53 on dist-test-slave-14jq"
server_key: "9ac6121b9957e6984e5690618e94771c"
server_key_iv: "986ea73cc44c36574ded5148486aa0f0"
server_key_version: "encryptionkey@0"
I20260927 15:20:53.750204 12381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:53.779322 12381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:53.780544 12381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:53.780934 12381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:53.804986 12381 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33565
I20260927 15:20:53.805032 12446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33565 every 8 connection(s)
I20260927 15:20:53.806866 12381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:53.811403 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12381
I20260927 15:20:53.811669 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:53.811689 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:53.813153 3033 external_mini_cluster.cc:1428] Setting key b0ec3831b37dccb2647cba4ba4be5d36
I20260927 15:20:53.824538 12447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec: Bootstrap starting.
I20260927 15:20:53.828450 12447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:53.829808 12447 log.cc:824] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:53.831636 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:53.816172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.2.246.126:33565)
0927 15:20:53.817750 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:53.817767 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:53.817827 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:20:53.819951 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:53.819990 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:53.820674 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:53.821174 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:53.822010 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.823444 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.825472 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:53.826503 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:53.826690 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:53.828048 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:53.828057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:53.828078 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:53.828875 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:53.829019 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:53.829046 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:53.829155 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:53.829444 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:20:53.829947 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":555,"threads_started":1}
I20260927 15:20:53.833976 12447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec: No bootstrap required, opened a new log
I20260927 15:20:53.842247 12447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } }
I20260927 15:20:53.842707 12447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:53.842778 12447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4facc2bf2b2940b4bec6dae9205a3fec, State: Initialized, Role: FOLLOWER
I20260927 15:20:53.843330 12447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } }
I20260927 15:20:53.843564 12447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:53.843659 12447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:53.843788 12447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:53.846127 12447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } }
I20260927 15:20:53.846700 12447 leader_election.cc:302] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4facc2bf2b2940b4bec6dae9205a3fec; no voters:
I20260927 15:20:53.847397 12447 leader_election.cc:288] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:53.847704 12452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:53.848562 12452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [term 1 LEADER]: Becoming Leader. State: Replica: 4facc2bf2b2940b4bec6dae9205a3fec, State: Running, Role: LEADER
I20260927 15:20:53.849218 12452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } }
I20260927 15:20:53.849745 12447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:53.853106 12454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4facc2bf2b2940b4bec6dae9205a3fec. Latest consensus state: current_term: 1 leader_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } } }
I20260927 15:20:53.853554 12454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:53.854264 12453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4facc2bf2b2940b4bec6dae9205a3fec" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33565 } } }
I20260927 15:20:53.855140 12453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:53.865029 12461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:53.870324 12461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:53.879699 12461 catalog_manager.cc:1347] Generated new cluster ID: 758713bcb97a4707b34cc8d871f965cd
I20260927 15:20:53.879840 12461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:53.909874 12461 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:53.911588 12461 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:53.919636 12461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4facc2bf2b2940b4bec6dae9205a3fec: Generated new TSK 0
I20260927 15:20:53.921181 12461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:53.939764 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--builtin_ntp_servers=127.2.246.84:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:20:54.290742 12471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:54.291085 12471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:54.291234 12471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:54.298872 12471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:54.299017 12471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:54.299168 12471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:54.308674 12471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:54.310071 12471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:54.311949 12471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:54.326449 12476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.326550 12477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.326849 12479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:54.328253 12471 server_base.cc:1034] running on GCE node
I20260927 15:20:54.328934 12471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:54.330171 12471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:54.331444 12471 hybrid_clock.cc:648] HybridClock initialized: now 1790522454331399 us; error 48 us; skew 500 ppm
I20260927 15:20:54.331789 12471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:54.334401 12471 webserver.cc:460] Webserver started at http://127.2.246.65:34523/ using document root <none> and password file <none>
I20260927 15:20:54.335150 12471 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:54.335292 12471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:54.335695 12471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:54.338682 12471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "ac1831d41f0d488095b99d1d5c892424"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "8da62a87c40f2c586d3ab5bd826ff511"
server_key_iv: "8cab9f1a149e2f6c92fee495ad7fde96"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.339517 12471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "ac1831d41f0d488095b99d1d5c892424"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "8da62a87c40f2c586d3ab5bd826ff511"
server_key_iv: "8cab9f1a149e2f6c92fee495ad7fde96"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.345199 12471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:20:54.349520 12485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.351598 12471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 15:20:54.351828 12471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "ac1831d41f0d488095b99d1d5c892424"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "8da62a87c40f2c586d3ab5bd826ff511"
server_key_iv: "8cab9f1a149e2f6c92fee495ad7fde96"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.352094 12471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:54.364930 12471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:54.365936 12471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:54.366259 12471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:54.367323 12471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:54.368970 12471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:54.369108 12471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.369252 12471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:54.369349 12471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.410156 12471 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:40063
I20260927 15:20:54.410317 12597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:40063 every 8 connection(s)
I20260927 15:20:54.411671 12471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:54.413254 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12471
I20260927 15:20:54.413452 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:54.414139 3033 external_mini_cluster.cc:1428] Setting key a78c00adee25067247109f97a845df3b
I20260927 15:20:54.417722 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--builtin_ntp_servers=127.2.246.84:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:20:54.428138 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.415444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:52325 (local address 127.2.246.126:33565)
0927 15:20:54.416495 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:54.416503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:54.417606 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 15:20:54.420107 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:54.420114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:54.420237 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:54.420399 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:54.421922 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.422715 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.424719 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.425188 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.425345 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.427147 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:54.427152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:54.427156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:54.427182 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:54.427227 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:54.427240 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:54.427346 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:54.427609 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 15:20:54.427736 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":715,"threads_started":1}
I20260927 15:20:54.430014 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.415991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33565 (local address 127.2.246.65:52325)
0927 15:20:54.417418 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:54.417469 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:54.419720 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:54.420716 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:54.420727 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:54.420864 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:54.421661 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:54.421683 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.422908 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.422913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:54.424546 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:54.424562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.425473 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.425482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:54.425768 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.426733 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:54.426735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:54.426787 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:54.426894 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:54.427342 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:54.427353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:54.427365 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:20:54.427605 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 15:20:54.428161 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":500,"threads_started":1}
I20260927 15:20:54.433046 12598 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33565
I20260927 15:20:54.433813 12598 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:54.435320 12598 heartbeater.cc:511] Master 127.2.246.126:33565 requested a full tablet report, sending...
I20260927 15:20:54.439244 12412 ts_manager.cc:194] Registered new tserver with Master: ac1831d41f0d488095b99d1d5c892424 (127.2.246.65:40063)
I20260927 15:20:54.441672 12412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.65:52325
W20260927 15:20:54.759341 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:54.759667 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:54.759797 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:54.767307 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:54.767426 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:54.767534 12602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:54.776774 12602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:54.778070 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:54.779817 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:54.793913 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.793955 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.794265 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:54.795192 12602 server_base.cc:1034] running on GCE node
I20260927 15:20:54.796052 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:54.797398 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:54.798702 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790522454798638 us; error 61 us; skew 500 ppm
I20260927 15:20:54.799149 12602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:54.802523 12602 webserver.cc:460] Webserver started at http://127.2.246.66:43227/ using document root <none> and password file <none>
I20260927 15:20:54.803503 12602 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:54.803660 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:54.804064 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:54.808384 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "566b0f79b02d458e903249ed8687bb80"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "93b728747a74efa33a2def45522ce686"
server_key_iv: "fb358b1db71cf872c05a28e39e6a9bf8"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.809470 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "566b0f79b02d458e903249ed8687bb80"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "93b728747a74efa33a2def45522ce686"
server_key_iv: "fb358b1db71cf872c05a28e39e6a9bf8"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.816452 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:20:54.821069 12616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.823418 12602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:20:54.823729 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "566b0f79b02d458e903249ed8687bb80"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-14jq"
server_key: "93b728747a74efa33a2def45522ce686"
server_key_iv: "fb358b1db71cf872c05a28e39e6a9bf8"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.824054 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:54.842752 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:54.843933 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:54.844313 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:54.845546 12602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:54.847667 12602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:54.847810 12602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.847973 12602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:54.848065 12602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.892944 12602 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:36363
I20260927 15:20:54.893018 12728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:36363 every 8 connection(s)
I20260927 15:20:54.894744 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:54.904146 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12602
I20260927 15:20:54.904394 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:54.904981 3033 external_mini_cluster.cc:1428] Setting key b99d025e505ec5891007c56f7806ccac
I20260927 15:20:54.908547 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--builtin_ntp_servers=127.2.246.84:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:20:54.910161 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.897703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40821 (local address 127.2.246.126:33565)
0927 15:20:54.897987 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:54.898004 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:54.899892 (+ 1888us) server_negotiation.cc:407] Connection header received
0927 15:20:54.902051 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:54.902056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:54.902136 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:54.902258 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:54.903850 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.904725 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.907123 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.907531 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.907678 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.909323 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:54.909329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:54.909332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:54.909351 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:54.909382 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:54.909394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:54.909495 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:54.909725 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:20:54.909825 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:20:54.911298 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.898292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33565 (local address 127.2.246.66:40821)
0927 15:20:54.899593 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:54.899695 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:54.901613 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:54.902593 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:54.902603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:54.902753 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:54.903600 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:54.903633 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.904909 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.904916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:54.906948 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:54.906964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.907837 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.907845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:54.908119 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.908956 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:54.908959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:54.909001 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:54.909089 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:54.909491 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:54.909501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:54.909522 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:20:54.909718 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 15:20:54.910182 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":384,"threads_started":1}
I20260927 15:20:54.912981 12729 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33565
I20260927 15:20:54.913352 12729 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:54.914448 12729 heartbeater.cc:511] Master 127.2.246.126:33565 requested a full tablet report, sending...
I20260927 15:20:54.916006 12412 ts_manager.cc:194] Registered new tserver with Master: 566b0f79b02d458e903249ed8687bb80 (127.2.246.66:36363)
I20260927 15:20:54.916879 12412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.66:40821
W20260927 15:20:55.347718 12732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:55.348133 12732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:55.348294 12732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:55.357410 12732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:55.357599 12732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:55.357790 12732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:55.370580 12732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:55.372296 12732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:55.374516 12732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:55.390450 12738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:55.390542 12740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:55.390491 12737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:55.391880 12732 server_base.cc:1034] running on GCE node
I20260927 15:20:55.392670 12732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:55.393949 12732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:55.395219 12732 hybrid_clock.cc:648] HybridClock initialized: now 1790522455395140 us; error 74 us; skew 500 ppm
I20260927 15:20:55.395610 12732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:55.398393 12732 webserver.cc:460] Webserver started at http://127.2.246.67:39067/ using document root <none> and password file <none>
I20260927 15:20:55.399228 12732 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:55.399508 12732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:55.399938 12732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:55.403577 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "9048cd32b9d24c11985e8039a668aec5"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-14jq"
server_key: "cf3b4fa88a1a6bd35119da08f46ba671"
server_key_iv: "3cbc4b46d17166469ad7cf1bb271b07b"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.404474 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "9048cd32b9d24c11985e8039a668aec5"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-14jq"
server_key: "cf3b4fa88a1a6bd35119da08f46ba671"
server_key_iv: "3cbc4b46d17166469ad7cf1bb271b07b"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.411496 12732 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 15:20:55.416352 12746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:55.418308 12732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 15:20:55.418562 12732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "9048cd32b9d24c11985e8039a668aec5"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-14jq"
server_key: "cf3b4fa88a1a6bd35119da08f46ba671"
server_key_iv: "3cbc4b46d17166469ad7cf1bb271b07b"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.418850 12732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:55.436203 12732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:55.437372 12732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:55.437733 12732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:55.439168 12732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:55.441334 12732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:55.441454 12732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:55.441584 12732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:55.441668 12732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:55.446055 12598 heartbeater.cc:503] Master 127.2.246.126:33565 was elected leader, sending a full tablet report...
I20260927 15:20:55.485875 12732 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:34565
I20260927 15:20:55.485948 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:34565 every 8 connection(s)
I20260927 15:20:55.487509 12732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:55.494309 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12732
I20260927 15:20:55.494513 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:55.495101 3033 external_mini_cluster.cc:1428] Setting key e5116582a03041f97b33f022de418c5b
I20260927 15:20:55.503098 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.490634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38699 (local address 127.2.246.126:33565)
0927 15:20:55.491226 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.491231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.492746 (+ 1515us) server_negotiation.cc:407] Connection header received
0927 15:20:55.494883 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.494890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.495003 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.495186 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.496797 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.497883 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.499721 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.500235 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.500371 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:55.502202 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.502208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.502212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.502234 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.502283 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.502304 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.502399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.502694 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.502796 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260927 15:20:55.504500 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.491361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33565 (local address 127.2.246.67:38699)
0927 15:20:55.492479 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:55.492584 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:55.494447 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:55.495512 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:55.495528 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:55.495629 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:55.496461 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:55.496481 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.498063 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.498067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:55.499545 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:55.499558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.500481 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.500494 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:55.500705 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:55.501673 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:55.501676 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:55.501733 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:55.501927 (+ 194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:55.502432 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:55.502442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:55.502450 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:20:55.502658 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:20:55.503125 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":320,"threads_started":1}
I20260927 15:20:55.506453 12859 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33565
I20260927 15:20:55.506871 12859 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:55.507781 12859 heartbeater.cc:511] Master 127.2.246.126:33565 requested a full tablet report, sending...
I20260927 15:20:55.509670 12412 ts_manager.cc:194] Registered new tserver with Master: 9048cd32b9d24c11985e8039a668aec5 (127.2.246.67:34565)
I20260927 15:20:55.510922 12412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.67:38699
I20260927 15:20:55.511989 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:20:55.519706 12861 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:55.517623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.2.246.126:33565)
0927 15:20:55.517897 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.517904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.518434 (+ 530us) server_negotiation.cc:407] Connection header received
0927 15:20:55.518596 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.518600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.518678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.518818 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.519350 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47484: BlockingRecv error: recv got EOF from 127.0.0.1:47484 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:20:55.520684 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12471
I20260927 15:20:55.530375 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.522120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.2.246.65:40063)
0927 15:20:55.522749 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.522756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.522788 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:20:55.522850 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.522857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.523083 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.523221 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.523744 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.524905 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.526542 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.527116 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.527233 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:55.528652 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.528660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.528674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.529419 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.529486 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.529508 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.529608 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.529817 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.529980 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
I20260927 15:20:55.531939 12593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:55.624719 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12471
I20260927 15:20:55.642359 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12602
I20260927 15:20:55.651607 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.643393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.2.246.66:36363)
0927 15:20:55.644051 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.644059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.644097 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:20:55.644166 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.644172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.644388 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.644554 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.645038 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.646135 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.647580 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.648147 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.648282 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:55.649751 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.649770 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.649786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.650735 (+ 949us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.650787 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.650830 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.650964 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.651139 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.651258 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":378,"threads_started":1}
I20260927 15:20:55.653378 12724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:55.737970 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12602
I20260927 15:20:55.755427 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12732
I20260927 15:20:55.765425 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.756573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.2.246.67:34565)
0927 15:20:55.757346 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.757355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.757400 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:20:55.757478 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.757487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.757780 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.758042 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.758544 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.759744 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.761111 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:55.762022 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:55.762184 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:55.763607 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.763618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.763638 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.764402 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.764462 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.764492 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.764637 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.764804 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.764986 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260927 15:20:55.767175 12854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:55.855062 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12732
I20260927 15:20:55.873757 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12381
I20260927 15:20:55.875452 12442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:55.983566 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12381
2026-09-27T15:20:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2717 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:56Z Disabled control of system clock
I20260927 15:20:56.069317 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35933
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:20:56.473654 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:56.474061 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:56.474231 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:56.481609 12879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:56.481734 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:56.481806 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:56.481953 12879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:56.482049 12879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:56.491508 12879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:56.492985 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:56.495201 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:56.508945 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:56.509001 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:56.509024 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:56.510138 12879 server_base.cc:1034] running on GCE node
I20260927 15:20:56.511031 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:56.512658 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:56.513963 12879 hybrid_clock.cc:648] HybridClock initialized: now 1790522456513905 us; error 63 us; skew 500 ppm
I20260927 15:20:56.514361 12879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:56.517046 12879 webserver.cc:460] Webserver started at http://127.2.246.126:34879/ using document root <none> and password file <none>
I20260927 15:20:56.517809 12879 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:56.518007 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:56.518350 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:56.521276 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "912774cbf3da42a49406ecbcce8020fd"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-14jq"
server_key: "0a68ba7e71ec74d6392591e4cad4b8e5"
server_key_iv: "e19e5820ccf5f5415302d8e1ff4ec3cc"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.522152 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "912774cbf3da42a49406ecbcce8020fd"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-14jq"
server_key: "0a68ba7e71ec74d6392591e4cad4b8e5"
server_key_iv: "e19e5820ccf5f5415302d8e1ff4ec3cc"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.528066 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:56.532047 12893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:56.533708 12879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:56.533951 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "912774cbf3da42a49406ecbcce8020fd"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-14jq"
server_key: "0a68ba7e71ec74d6392591e4cad4b8e5"
server_key_iv: "e19e5820ccf5f5415302d8e1ff4ec3cc"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.534184 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:56.540338 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:56.541191 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:56.541559 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:56.562516 12879 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:35933
I20260927 15:20:56.562690 12944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:35933 every 8 connection(s)
I20260927 15:20:56.564126 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:56.567992 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12879
I20260927 15:20:56.568264 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:20:56.569131 3033 external_mini_cluster.cc:1428] Setting key 204290545bc65efc130fbbcee0fe92cf
I20260927 15:20:56.569749 12945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:56.580251 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:56.571860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.2.246.126:35933)
0927 15:20:56.573167 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:56.573185 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:56.573239 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:20:56.575360 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:56.575389 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:56.576072 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:56.576631 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:56.578298 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:56.578305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:56.578322 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:56.578339 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:56.578427 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:56.578448 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:56.578753 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:56.578877 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:20:56.579239 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":456,"threads_started":1}
I20260927 15:20:56.586005 12945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd: Bootstrap starting.
I20260927 15:20:56.589744 12945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:56.591143 12945 log.cc:824] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:56.594871 12945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd: No bootstrap required, opened a new log
I20260927 15:20:56.603436 12945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } }
I20260927 15:20:56.603976 12945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:56.604087 12945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912774cbf3da42a49406ecbcce8020fd, State: Initialized, Role: FOLLOWER
I20260927 15:20:56.604830 12945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } }
I20260927 15:20:56.605094 12945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:56.605257 12945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:56.605450 12945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:56.607981 12945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } }
I20260927 15:20:56.608620 12945 leader_election.cc:302] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 912774cbf3da42a49406ecbcce8020fd; no voters:
I20260927 15:20:56.609516 12945 leader_election.cc:288] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:56.609658 12950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:56.610752 12950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [term 1 LEADER]: Becoming Leader. State: Replica: 912774cbf3da42a49406ecbcce8020fd, State: Running, Role: LEADER
I20260927 15:20:56.611542 12950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } }
I20260927 15:20:56.612372 12945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:56.615079 12952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 912774cbf3da42a49406ecbcce8020fd. Latest consensus state: current_term: 1 leader_uuid: "912774cbf3da42a49406ecbcce8020fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } } }
I20260927 15:20:56.615697 12952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:56.615203 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "912774cbf3da42a49406ecbcce8020fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912774cbf3da42a49406ecbcce8020fd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35933 } } }
I20260927 15:20:56.616434 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:56.617684 12955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:56.624373 12955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:56.636322 12955 catalog_manager.cc:1347] Generated new cluster ID: f6f1b63d8e0b467ea640ec12dff6bbac
I20260927 15:20:56.636498 12955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:56.674042 12955 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:56.674664 12955 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:56.685120 12955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 912774cbf3da42a49406ecbcce8020fd: Generated new TSK 0
I20260927 15:20:56.686745 12955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:56.707181 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--builtin_ntp_servers=127.2.246.84:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:20:57.129930 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:57.130312 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:57.130419 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:57.139081 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:57.139231 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:57.139362 12969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:20:57.150534 12969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:57.152253 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:57.154443 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:57.171665 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:57.171569 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:57.171720 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:57.172403 12969 server_base.cc:1034] running on GCE node
I20260927 15:20:57.173318 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:57.174573 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:57.175859 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790522457175797 us; error 59 us; skew 500 ppm
I20260927 15:20:57.176301 12969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:57.183138 12969 webserver.cc:460] Webserver started at http://127.2.246.65:37321/ using document root <none> and password file <none>
I20260927 15:20:57.184064 12969 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:57.184229 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:57.184594 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:57.187881 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "c313827d57d443acb77eaa0f8c9338d8"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "d3a598dc147aaddfaadd88ca4ecb516f"
server_key_iv: "32c6e8987cb724d708d28d7f2d3ff04e"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.188752 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "c313827d57d443acb77eaa0f8c9338d8"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "d3a598dc147aaddfaadd88ca4ecb516f"
server_key_iv: "32c6e8987cb724d708d28d7f2d3ff04e"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.194819 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:20:57.199282 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.201225 12969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:20:57.201462 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c313827d57d443acb77eaa0f8c9338d8"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "d3a598dc147aaddfaadd88ca4ecb516f"
server_key_iv: "32c6e8987cb724d708d28d7f2d3ff04e"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.201742 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:57.208066 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:57.208905 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:57.209188 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:57.210275 12969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:57.211871 12969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:57.211994 12969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.212141 12969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:57.212227 12969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.253305 12969 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45679
I20260927 15:20:57.253410 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45679 every 8 connection(s)
I20260927 15:20:57.254851 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:57.258953 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 12969
I20260927 15:20:57.259222 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:57.260118 3033 external_mini_cluster.cc:1428] Setting key f98fb2f63e5087f580f7a2e064e17b45
I20260927 15:20:57.263916 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--builtin_ntp_servers=127.2.246.84:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:20:57.266314 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:57.258564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:46067 (local address 127.2.246.126:35933)
0927 15:20:57.259055 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:57.259060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:57.261469 (+ 2409us) server_negotiation.cc:407] Connection header received
0927 15:20:57.263651 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:57.263657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:57.263758 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:57.263882 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:57.265459 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:57.265465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:57.265470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:57.265491 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:57.265543 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:57.265557 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:57.265696 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:57.265919 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 15:20:57.265992 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":289,"threads_started":1}
I20260927 15:20:57.267406 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:57.259719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35933 (local address 127.2.246.65:46067)
0927 15:20:57.261279 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:57.261350 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:57.263306 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:57.264201 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:57.264214 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:57.264347 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:57.265140 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:57.265142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:57.265185 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:57.265275 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:57.265654 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:57.265664 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:57.265672 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:20:57.266022 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 15:20:57.266342 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":513,"threads_started":1}
I20260927 15:20:57.269999 13096 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35933
I20260927 15:20:57.270447 13096 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:57.271158 13096 heartbeater.cc:511] Master 127.2.246.126:35933 requested a full tablet report, sending...
I20260927 15:20:57.274654 12910 ts_manager.cc:194] Registered new tserver with Master: c313827d57d443acb77eaa0f8c9338d8 (127.2.246.65:45679)
W20260927 15:20:57.625326 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:57.625666 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:57.625806 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:57.633980 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:57.634091 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:57.634263 13100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:20:57.643941 13100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:57.645370 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:57.647147 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:57.662132 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:57.662187 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:57.662243 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:57.663457 13100 server_base.cc:1034] running on GCE node
I20260927 15:20:57.664325 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:57.665575 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:57.666882 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790522457666830 us; error 69 us; skew 500 ppm
I20260927 15:20:57.667290 13100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:57.670603 13100 webserver.cc:460] Webserver started at http://127.2.246.66:44489/ using document root <none> and password file <none>
I20260927 15:20:57.671439 13100 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:57.671552 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:57.671864 13100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:57.675835 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "ae4132da8ebd4337b15e04d05ff32743"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "ff260e8af2c2ae1f8a5730624ba8fe02"
server_key_iv: "b16bc9879d11c757cb209a7119e8df7a"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.676815 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "ae4132da8ebd4337b15e04d05ff32743"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "ff260e8af2c2ae1f8a5730624ba8fe02"
server_key_iv: "b16bc9879d11c757cb209a7119e8df7a"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.683831 13100 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 15:20:57.688789 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.691088 13100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:20:57.691366 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "ae4132da8ebd4337b15e04d05ff32743"
format_stamp: "Formatted at 2026-09-27 15:20:57 on dist-test-slave-14jq"
server_key: "ff260e8af2c2ae1f8a5730624ba8fe02"
server_key_iv: "b16bc9879d11c757cb209a7119e8df7a"
server_key_version: "encryptionkey@0"
I20260927 15:20:57.691689 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:57.699393 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:57.700579 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:57.700929 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:57.702241 13100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:57.704144 13100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:57.704284 13100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.704437 13100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:57.704531 13100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:57.748422 13100 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:45933
I20260927 15:20:57.748498 13226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:45933 every 8 connection(s)
I20260927 15:20:57.750310 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:57.756114 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13100
I20260927 15:20:57.756280 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:57.756721 3033 external_mini_cluster.cc:1428] Setting key d50c24a0d8e88435a07d1a486182d428
I20260927 15:20:57.760565 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--builtin_ntp_servers=127.2.246.84:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:20:57.761292 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:57.753608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:33731 (local address 127.2.246.126:35933)
0927 15:20:57.753882 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:57.753889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:57.755751 (+ 1862us) server_negotiation.cc:407] Connection header received
0927 15:20:57.758350 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:57.758356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:57.758442 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:57.758574 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:57.760308 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:57.760314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:57.760326 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:57.760349 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:57.760396 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:57.760411 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:57.760571 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:57.760864 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:20:57.760935 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 15:20:57.763262 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:57.754137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35933 (local address 127.2.246.66:33731)
0927 15:20:57.755441 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:57.755505 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:57.757817 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:57.758842 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:57.758852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:57.759015 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:57.759872 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:57.759877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:57.759931 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:57.760061 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:57.760573 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:57.760585 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:57.760596 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:20:57.761622 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0927 15:20:57.761957 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":361,"threads_started":1}
I20260927 15:20:57.764936 13227 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35933
I20260927 15:20:57.765435 13227 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:57.766435 13227 heartbeater.cc:511] Master 127.2.246.126:35933 requested a full tablet report, sending...
I20260927 15:20:57.768543 12910 ts_manager.cc:194] Registered new tserver with Master: ae4132da8ebd4337b15e04d05ff32743 (127.2.246.66:45933)
W20260927 15:20:58.175465 13230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:58.175938 13230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:58.176108 13230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:58.186410 13230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:58.186617 13230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:58.186827 13230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:20:58.199957 13230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:20:58.201699 13230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:58.204018 13230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:58.220453 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:58.220920 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:58.221153 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:58.222121 13230 server_base.cc:1034] running on GCE node
I20260927 15:20:58.222851 13230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:58.224051 13230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:58.225334 13230 hybrid_clock.cc:648] HybridClock initialized: now 1790522458225287 us; error 77 us; skew 500 ppm
I20260927 15:20:58.225732 13230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:20:58.228319 13230 webserver.cc:460] Webserver started at http://127.2.246.67:41069/ using document root <none> and password file <none>
I20260927 15:20:58.229137 13230 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:58.229282 13230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:58.229620 13230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:58.232786 13230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "887f3c2039f3420eb908db9eb724feef"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-14jq"
server_key: "19272de0030d061b1e0c0498c069b530"
server_key_iv: "fa1df59c014c6f71c69372f387ed88b7"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.233588 13230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "887f3c2039f3420eb908db9eb724feef"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-14jq"
server_key: "19272de0030d061b1e0c0498c069b530"
server_key_iv: "fa1df59c014c6f71c69372f387ed88b7"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.239390 13230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:20:58.243296 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:58.245138 13230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:58.245363 13230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "887f3c2039f3420eb908db9eb724feef"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-14jq"
server_key: "19272de0030d061b1e0c0498c069b530"
server_key_iv: "fa1df59c014c6f71c69372f387ed88b7"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.245616 13230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:58.252008 13230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:58.253050 13230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:58.253342 13230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:58.254495 13230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:58.256167 13230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:58.256292 13230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:58.256426 13230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:58.256521 13230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:58.279203 13096 heartbeater.cc:503] Master 127.2.246.126:35933 was elected leader, sending a full tablet report...
I20260927 15:20:58.316579 13230 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:38051
I20260927 15:20:58.316726 13356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:38051 every 8 connection(s)
I20260927 15:20:58.318152 13230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:58.323342 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13230
I20260927 15:20:58.323622 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:58.324308 3033 external_mini_cluster.cc:1428] Setting key 330d07ca29272c3134262eb2ea439f1a
I20260927 15:20:58.330233 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.321594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:45237 (local address 127.2.246.126:35933)
0927 15:20:58.322199 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.322205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.324087 (+ 1882us) server_negotiation.cc:407] Connection header received
0927 15:20:58.326985 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.326991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.327099 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.327257 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.329253 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.329267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.329270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.329291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.329339 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.329352 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.329483 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.329697 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.329766 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":358,"threads_started":1}
I20260927 15:20:58.331691 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.322206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:35933 (local address 127.2.246.67:45237)
0927 15:20:58.323811 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:58.323879 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:58.326551 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:58.327582 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:58.327593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:58.327825 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:58.328878 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:58.328880 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:58.328933 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:58.329048 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:58.329455 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:58.329465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:58.329473 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:20:58.329785 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 15:20:58.330252 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":499,"threads_started":1}
I20260927 15:20:58.333633 13357 heartbeater.cc:347] Connected to a master server at 127.2.246.126:35933
I20260927 15:20:58.334065 13357 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:58.334787 13357 heartbeater.cc:511] Master 127.2.246.126:35933 requested a full tablet report, sending...
I20260927 15:20:58.336495 12910 ts_manager.cc:194] Registered new tserver with Master: 887f3c2039f3420eb908db9eb724feef (127.2.246.67:38051)
I20260927 15:20:58.341292 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:20:58.349545 13359 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:58.347856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.2.246.126:35933)
0927 15:20:58.348236 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.348243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.348405 (+ 162us) server_negotiation.cc:407] Connection header received
0927 15:20:58.348594 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.348598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.348658 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.348775 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.349287 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39128: BlockingRecv error: recv got EOF from 127.0.0.1:39128 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
I20260927 15:20:58.350342 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12969
I20260927 15:20:58.355051 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.352097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.2.246.65:45679)
0927 15:20:58.352754 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.352762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.352796 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:20:58.352880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.352888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.353165 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.353322 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.354344 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.354352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.354365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.354377 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.354423 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.354438 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.354591 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.354643 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.354740 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":293,"threads_started":1}
I20260927 15:20:58.356520 13091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:58.448836 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12969
I20260927 15:20:58.470090 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13100
I20260927 15:20:58.474628 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.471410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.2.246.66:45933)
0927 15:20:58.471991 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.471998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.472026 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:20:58.472090 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.472098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.472317 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.472459 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.473575 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.473586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.473604 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.473643 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.473717 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.473738 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.474053 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.474273 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.474373 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260927 15:20:58.476027 13222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:58.568557 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13100
I20260927 15:20:58.587858 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13230
I20260927 15:20:58.592793 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.589022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.2.246.67:38051)
0927 15:20:58.589821 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.589880 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.589954 (+ 74us) server_negotiation.cc:407] Connection header received
0927 15:20:58.590039 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.590047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.590358 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.590566 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.591929 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.591936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.591950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.591970 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.592015 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.592031 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.592244 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.592325 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.592448 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":384,"threads_started":1}
I20260927 15:20:58.594501 13352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:58.686353 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13230
I20260927 15:20:58.705132 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 12879
I20260927 15:20:58.706715 12940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:20:58.810794 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 12879
2026-09-27T15:20:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2823 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:58 dist-test-slave-14jq krb5kdc[13376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:58 dist-test-slave-14jq krb5kdc[13376](info): set up 2 sockets
Sep 27 15:20:58 dist-test-slave-14jq krb5kdc[13376](info): commencing operation
krb5kdc: starting...
W20260927 15:20:59.975508 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:21:00 dist-test-slave-14jq krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:21:00.275163 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39533
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:00.682978 13389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:00.683301 13389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:00.683604 13389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:00.690928 13389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:00.691056 13389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:00.691102 13389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:00.691151 13389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:00.691187 13389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:00.701625 13389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:39533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:00.703081 13389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:00.704401 13389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:00.719244 13397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:00.719460 13394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:00.719509 13395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:00.720818 13389 server_base.cc:1034] running on GCE node
I20260927 15:21:00.722044 13389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:00.724017 13389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:00.725270 13389 hybrid_clock.cc:648] HybridClock initialized: now 1790522460725186 us; error 75 us; skew 500 ppm
I20260927 15:21:00.725744 13389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:00 dist-test-slave-14jq krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:00.729733 13389 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:21:00.732113 13389 webserver.cc:460] Webserver started at http://127.2.246.126:33379/ using document root <none> and password file <none>
I20260927 15:21:00.733258 13389 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:00.733433 13389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:00.733772 13389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:00.737955 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "b844f3f1c69741a9b07f54bbe2b94f79"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-14jq"
server_key: "6d003bde8de333fa85d25d1a7b3ddf5b"
server_key_iv: "bb91c117b7caaf18ff627e2cae2ad75a"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.739313 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "b844f3f1c69741a9b07f54bbe2b94f79"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-14jq"
server_key: "6d003bde8de333fa85d25d1a7b3ddf5b"
server_key_iv: "bb91c117b7caaf18ff627e2cae2ad75a"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.747381 13389 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 15:21:00.752254 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:00.754401 13389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:21:00.754678 13389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "b844f3f1c69741a9b07f54bbe2b94f79"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-14jq"
server_key: "6d003bde8de333fa85d25d1a7b3ddf5b"
server_key_iv: "bb91c117b7caaf18ff627e2cae2ad75a"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.754953 13389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:00.770632 13389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:00.777010 13389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:00.777384 13389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:00.801899 13389 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:39533
I20260927 15:21:00.801982 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:39533 every 8 connection(s)
I20260927 15:21:00.803674 13389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:00.806525 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13389
I20260927 15:21:00.806788 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:21:00.807513 3033 external_mini_cluster.cc:1428] Setting key 472a11f4a7c919d0aff877305117f571
I20260927 15:21:00.808315 13456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:00.822914 13456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: Bootstrap starting.
Sep 27 15:21:00 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:21:00.826408 13456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:00.827595 13456 log.cc:824] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:00.830797 13456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: No bootstrap required, opened a new log
I20260927 15:21:00.835604 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:00.810133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.2.246.126:39533)
0927 15:21:00.811102 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:00.811119 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:00.811179 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:21:00.813474 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:00.813504 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:00.814092 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:00.814600 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:00.816431 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.818339 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.819767 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.820881 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.821087 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:00.825504 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:00.825515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:00.825534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:00.825585 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:00.829561 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.830563 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.830572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.830580 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.830803 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.831238 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.831255 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.831258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.832118 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:00.832543 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:00.833229 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 15:21:00.833810 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":417,"threads_started":1}
I20260927 15:21:00.840386 13456 raft_consensus.cc:348] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } }
I20260927 15:21:00.840838 13456 raft_consensus.cc:374] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:00.840955 13456 raft_consensus.cc:729] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b844f3f1c69741a9b07f54bbe2b94f79, State: Initialized, Role: FOLLOWER
I20260927 15:21:00.841595 13456 consensus_queue.cc:260] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } }
I20260927 15:21:00.841849 13456 raft_consensus.cc:388] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:21:00.842063 13456 raft_consensus.cc:482] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:21:00.842255 13456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:00.844521 13456 raft_consensus.cc:504] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } }
I20260927 15:21:00.845124 13456 leader_election.cc:302] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b844f3f1c69741a9b07f54bbe2b94f79; no voters:
I20260927 15:21:00.846089 13456 leader_election.cc:288] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:21:00.846259 13461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:00.847752 13461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [term 1 LEADER]: Becoming Leader. State: Replica: b844f3f1c69741a9b07f54bbe2b94f79, State: Running, Role: LEADER
I20260927 15:21:00.848502 13461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } }
I20260927 15:21:00.848946 13456 sys_catalog.cc:564] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:00.852172 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b844f3f1c69741a9b07f54bbe2b94f79. Latest consensus state: current_term: 1 leader_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } } }
I20260927 15:21:00.852725 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:00.853475 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b844f3f1c69741a9b07f54bbe2b94f79" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39533 } } }
I20260927 15:21:00.853796 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:00.856333 13471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:00.862658 13471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:00.873888 13471 catalog_manager.cc:1347] Generated new cluster ID: 454fecaf0a7e46e69cf6ed462fb4280f
I20260927 15:21:00.874063 13471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:00.895505 13471 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:00.897176 13471 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:00.906551 13471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: Generated new TSK 0
I20260927 15:21:00.907766 13471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:21:00.977542 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:01.408257 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:01.408640 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:01.408751 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:01.417695 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:01.417984 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:01.418241 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:21:01.430474 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:01.432282 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:01.434067 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:01.450800 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:01.450901 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:01.450845 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:01.451975 13482 server_base.cc:1034] running on GCE node
I20260927 15:21:01.452840 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:01.454316 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:01.455614 13482 hybrid_clock.cc:648] HybridClock initialized: now 1790522461455524 us; error 62 us; skew 500 ppm
I20260927 15:21:01.456046 13482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:01 dist-test-slave-14jq krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:01.460496 13482 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:21:01.462883 13482 webserver.cc:460] Webserver started at http://127.2.246.65:37777/ using document root <none> and password file <none>
I20260927 15:21:01.463884 13482 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:01.464027 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:01.464365 13482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:01.469377 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "a85350d1710447b4b6a0cc857e3ff3eb"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-14jq"
server_key: "289cdaffee72a51b933a648e6693fecc"
server_key_iv: "2c860d178a29d91865dcca91376d6541"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.470840 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "a85350d1710447b4b6a0cc857e3ff3eb"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-14jq"
server_key: "289cdaffee72a51b933a648e6693fecc"
server_key_iv: "2c860d178a29d91865dcca91376d6541"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.478706 13482 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 15:21:01.483642 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:01.485773 13482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:21:01.486403 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "a85350d1710447b4b6a0cc857e3ff3eb"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-14jq"
server_key: "289cdaffee72a51b933a648e6693fecc"
server_key_iv: "2c860d178a29d91865dcca91376d6541"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.486727 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:01.503965 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:01.510321 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:01.510710 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:01.512071 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:01.514014 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:01.514130 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:01.514297 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:01.514379 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:01.557971 13482 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:41037
I20260927 15:21:01.558084 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:41037 every 8 connection(s)
I20260927 15:21:01.559688 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:21:01.561477 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13482
I20260927 15:21:01.561748 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:21:01.562817 3033 external_mini_cluster.cc:1428] Setting key 02b6f0d5c4588f31b9104ea44cb9d4e6
Sep 27 15:21:01 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:21:01.587085 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:01.563705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:52211 (local address 127.2.246.126:39533)
0927 15:21:01.564226 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:01.564231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:01.565797 (+ 1566us) server_negotiation.cc:407] Connection header received
0927 15:21:01.568020 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:01.568029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:01.568150 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:01.568340 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:01.570313 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.571553 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.573519 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.574352 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.574531 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:01.579701 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:01.579708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:01.579712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:01.579760 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:01.583045 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:01.584181 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:01.584186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:01.584189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:01.584351 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:01.584863 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:01.584871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:01.584874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:01.585204 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:01.585384 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:01.586378 (+ 994us) server_negotiation.cc:299] Negotiation successful
0927 15:21:01.586666 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260927 15:21:01.588356 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:01.564525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39533 (local address 127.2.246.65:52211)
0927 15:21:01.565573 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:01.565637 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:01.567648 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:01.568632 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:01.568642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:01.569265 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:01.570052 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:01.570074 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.571836 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.571841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:01.573290 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:01.573306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.574658 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.574665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:01.574931 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:01.575920 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:01.575948 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:01.579337 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:01.583330 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:01.583346 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:01.583369 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:01.584000 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:01.584483 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:01.584488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:01.584492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:01.584724 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:01.585346 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:01.585356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:01.585902 (+ 546us) client_negotiation.cc:769] Sending connection context
0927 15:21:01.586253 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 15:21:01.586964 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":432,"threads_started":1}
I20260927 15:21:01.590826 13610 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39533
I20260927 15:21:01.591305 13610 heartbeater.cc:464] Registering TS with master...
I20260927 15:21:01.592372 13610 heartbeater.cc:511] Master 127.2.246.126:39533 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:21:01.595482 13421 ts_manager.cc:194] Registered new tserver with Master: a85350d1710447b4b6a0cc857e3ff3eb (127.2.246.65:41037)
I20260927 15:21:01.597204 13421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:52211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:21:01.630587 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:02.007655 13616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:02.007968 13616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:02.008136 13616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:02.015775 13616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:02.015920 13616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:02.016098 13616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:21:02.026305 13616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:02.027729 13616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:02.029155 13616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:02.044307 13621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:02.044273 13624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:02.044210 13622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:02.045136 13616 server_base.cc:1034] running on GCE node
I20260927 15:21:02.045974 13616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:02.047096 13616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:02.048448 13616 hybrid_clock.cc:648] HybridClock initialized: now 1790522462048395 us; error 49 us; skew 500 ppm
I20260927 15:21:02.048962 13616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:02 dist-test-slave-14jq krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:02.053488 13616 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:21:02.055534 13616 webserver.cc:460] Webserver started at http://127.2.246.66:42605/ using document root <none> and password file <none>
I20260927 15:21:02.056509 13616 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:02.056636 13616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:02.056939 13616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:02.060837 13616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "54acb8cca8294d51a2ea64d31b8e69b0"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "c58828be79cf3d6fe4cfcb3e2f20046b"
server_key_iv: "f5b66d8ab1efbdb5771fcfb9ec331b35"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.061733 13616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "54acb8cca8294d51a2ea64d31b8e69b0"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "c58828be79cf3d6fe4cfcb3e2f20046b"
server_key_iv: "f5b66d8ab1efbdb5771fcfb9ec331b35"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.068691 13616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:21:02.073771 13631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.075955 13616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:21:02.076244 13616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "54acb8cca8294d51a2ea64d31b8e69b0"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "c58828be79cf3d6fe4cfcb3e2f20046b"
server_key_iv: "f5b66d8ab1efbdb5771fcfb9ec331b35"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.076520 13616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:02.101359 13616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:02.108175 13616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:02.108628 13616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:02.110185 13616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:02.112092 13616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:02.112185 13616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.112358 13616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:02.112416 13616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.158656 13616 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:40995
I20260927 15:21:02.158707 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:40995 every 8 connection(s)
I20260927 15:21:02.160485 13616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:21:02.161154 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13616
I20260927 15:21:02.161379 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:21:02.162035 3033 external_mini_cluster.cc:1428] Setting key efa2029453e51745cee5e114050a2e41
Sep 27 15:21:02 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:21:02.196331 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.164222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:45025 (local address 127.2.246.126:39533)
0927 15:21:02.167432 (+ 3210us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.167437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.167460 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:21:02.170548 (+ 3088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.170553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.170681 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.170825 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:02.174132 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.174989 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.177818 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.178412 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.178576 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.183079 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:02.183085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:02.183089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:02.183130 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:02.191369 (+ 8239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.192652 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.192657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.192660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.192821 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.193457 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.193464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.193467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.193978 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:02.194147 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.195443 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.195724 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3088,"thread_start_us":2971,"threads_started":1}
I20260927 15:21:02.197773 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.165108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39533 (local address 127.2.246.66:45025)
0927 15:21:02.166797 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.166899 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.170049 (+ 3150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.171239 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.171257 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.172580 (+ 1323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:02.173758 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.173800 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.175242 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.175251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.177574 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.177594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.178727 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.178735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.179022 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.179897 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:02.179932 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:02.182731 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:02.191635 (+ 8904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:02.191646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:02.191668 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:02.192469 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:02.192956 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:02.192962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:02.192966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:02.193213 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:02.194285 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:02.194301 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:02.194834 (+ 533us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.195332 (+ 498us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.196058 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":491,"threads_started":1}
I20260927 15:21:02.200083 13744 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39533
I20260927 15:21:02.200560 13744 heartbeater.cc:464] Registering TS with master...
I20260927 15:21:02.201561 13744 heartbeater.cc:511] Master 127.2.246.126:39533 requested a full tablet report, sending...
I20260927 15:21:02.203843 13421 ts_manager.cc:194] Registered new tserver with Master: 54acb8cca8294d51a2ea64d31b8e69b0 (127.2.246.66:40995)
I20260927 15:21:02.205295 13421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:45025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:21:02.231278 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:02.578758 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:02.579116 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:02.579298 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:02.587121 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:02.587317 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:02.587493 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:21:02.597985 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:02.599372 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:02.600867 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:21:02.602559 13610 heartbeater.cc:503] Master 127.2.246.126:39533 was elected leader, sending a full tablet report...
W20260927 15:21:02.619942 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:02.620024 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:02.622030 13750 server_base.cc:1034] running on GCE node
W20260927 15:21:02.626080 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:02.626974 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:02.628176 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:02.629436 13750 hybrid_clock.cc:648] HybridClock initialized: now 1790522462629373 us; error 62 us; skew 500 ppm
I20260927 15:21:02.631405 13750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:02 dist-test-slave-14jq krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:02.634857 13750 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:21:02.636765 13750 webserver.cc:460] Webserver started at http://127.2.246.67:46031/ using document root <none> and password file <none>
I20260927 15:21:02.637578 13750 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:02.637730 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:02.638090 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:02.641582 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "566fbeae24de4c28ac3bee1285fd8931"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "f4089e5ceee761fd1a7fcaf039a427f5"
server_key_iv: "82df6303c93507f8cefc43ec07b39d71"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.642519 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "566fbeae24de4c28ac3bee1285fd8931"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "f4089e5ceee761fd1a7fcaf039a427f5"
server_key_iv: "82df6303c93507f8cefc43ec07b39d71"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.648803 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:21:02.652997 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.654582 13750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:21:02.654759 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "566fbeae24de4c28ac3bee1285fd8931"
format_stamp: "Formatted at 2026-09-27 15:21:02 on dist-test-slave-14jq"
server_key: "f4089e5ceee761fd1a7fcaf039a427f5"
server_key_iv: "82df6303c93507f8cefc43ec07b39d71"
server_key_version: "encryptionkey@0"
I20260927 15:21:02.654961 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:02.672343 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:02.676936 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:02.677243 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:02.678397 13750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:02.679929 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:02.680006 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.680137 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:02.680188 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:02.720736 13750 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:44993
I20260927 15:21:02.720794 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:44993 every 8 connection(s)
I20260927 15:21:02.722196 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:21:02.728046 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 13750
I20260927 15:21:02.728294 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:21:02.728770 3033 external_mini_cluster.cc:1428] Setting key de22b476c4cd4bd73055e0da138e0ddf
Sep 27 15:21:02 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:21:02.747102 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.725725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:41119 (local address 127.2.246.126:39533)
0927 15:21:02.726221 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.726225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.727632 (+ 1407us) server_negotiation.cc:407] Connection header received
0927 15:21:02.730301 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.730305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.730459 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.730602 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:02.732897 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.733795 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.735549 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.736017 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.736141 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.739818 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:02.739825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:02.739829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:02.739878 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:02.742922 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.743983 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.743990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.743994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.744165 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.744881 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.744886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.744891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.745222 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:02.745409 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.746359 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.746652 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260927 15:21:02.748647 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.726412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39533 (local address 127.2.246.67:41119)
0927 15:21:02.727386 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.727474 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.729532 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.730973 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.730988 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.731807 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:02.732587 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.732640 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.734054 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.734058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.735376 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.735388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.736262 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.736267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.736490 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.737253 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:02.737280 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:02.739506 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:02.743156 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:02.743166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:02.743185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:02.743804 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:02.744391 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:02.744397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:02.744401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:02.744731 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:02.745381 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:02.745392 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:02.745924 (+ 532us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.746261 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.746957 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":394,"threads_started":1}
I20260927 15:21:02.750520 13878 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39533
I20260927 15:21:02.751026 13878 heartbeater.cc:464] Registering TS with master...
I20260927 15:21:02.752105 13878 heartbeater.cc:511] Master 127.2.246.126:39533 requested a full tablet report, sending...
I20260927 15:21:02.754086 13421 ts_manager.cc:194] Registered new tserver with Master: 566fbeae24de4c28ac3bee1285fd8931 (127.2.246.67:44993)
I20260927 15:21:02.754951 13421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:41119
I20260927 15:21:02.758311 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:21:02.778949 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.764066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.2.246.126:39533)
0927 15:21:02.764396 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.764407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.764793 (+ 386us) server_negotiation.cc:407] Connection header received
0927 15:21:02.765101 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.765106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.765186 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.765329 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:02.767010 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.767838 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.769184 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.769627 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.769768 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.772689 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:02.772696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:02.772700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:02.772759 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:02.775872 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.776710 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.776718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.776727 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.776860 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:02.777275 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:02.777281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:02.777284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:02.777664 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:02.777975 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.778328 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.778582 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:21:02.784185 13421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:21:02.791198 13421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:21:02.836305 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.806361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.2.246.67:44993)
0927 15:21:02.821693 (+ 15332us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.821706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.823919 (+ 2213us) server_negotiation.cc:407] Connection header received
0927 15:21:02.824037 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.824046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.824369 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.824571 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:02.825691 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.827932 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.833278 (+ 5346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.834666 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.834851 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.835164 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.835627 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.835873 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15061,"thread_start_us":15338,"threads_started":1}
I20260927 15:21:02.836340 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.806129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:41037 (local address 127.0.0.1:57610)
0927 15:21:02.825339 (+ 19210us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.825363 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.825488 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.826524 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.826530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.826557 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:02.827023 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.827035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.829115 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.829125 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.833045 (+ 3920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.833063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.835343 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.835351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.835550 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.835578 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.835660 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.835784 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19064,"mutex_wait_us":95,"thread_start_us":16679,"threads_started":1}
I20260927 15:21:02.836815 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.806712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.2.246.65:41037)
0927 15:21:02.822628 (+ 15916us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.822638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.825622 (+ 2984us) server_negotiation.cc:407] Connection header received
0927 15:21:02.825779 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.825788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.826071 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.826258 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:02.827248 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.828879 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.833341 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.834909 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.835080 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.835355 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.835963 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.836197 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15709,"thread_start_us":15499,"threads_started":1}
I20260927 15:21:02.837108 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.805923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:44993 (local address 127.0.0.1:51968)
0927 15:21:02.823277 (+ 17354us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.823354 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.823663 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.824682 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.824694 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.824747 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:02.825421 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.825437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.828192 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.828201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.833045 (+ 4844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.833063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.834946 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.834953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.835240 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.835327 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.835504 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.835769 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17020,"mutex_wait_us":39,"thread_start_us":19108,"threads_started":1}
I20260927 15:21:02.840392 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.828507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40995 (local address 127.0.0.1:59516)
0927 15:21:02.830664 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.830689 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.830838 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.831630 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.831635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.831659 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:02.832284 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.832295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.833968 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.833975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.836151 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.836175 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.839552 (+ 3377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.839558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.839700 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.839727 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.839854 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.840027 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"thread_start_us":1825,"threads_started":1}
I20260927 15:21:02.840732 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.825737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.2.246.66:40995)
0927 15:21:02.831040 (+ 5303us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.831047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.831072 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:21:02.831170 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.831177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.831408 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.831654 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:02.832455 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.833745 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.836427 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.838993 (+ 2566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.839213 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.839457 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.839918 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.840105 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5114,"thread_start_us":4928,"threads_started":1}
I20260927 15:21:02.841598 13813 tablet_service.cc:1467] Processing CreateTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec (DEFAULT_TABLE table=test-table [id=822e2913999945d3ab8923903ef10c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:02.841589 13545 tablet_service.cc:1467] Processing CreateTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec (DEFAULT_TABLE table=test-table [id=822e2913999945d3ab8923903ef10c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:02.843212 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2866f2ecc6c47eb8e647a0d6cc6adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:02.843221 13813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2866f2ecc6c47eb8e647a0d6cc6adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:02.844637 13679 tablet_service.cc:1467] Processing CreateTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec (DEFAULT_TABLE table=test-table [id=822e2913999945d3ab8923903ef10c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:02.846159 13679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2866f2ecc6c47eb8e647a0d6cc6adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:02.856041 13897 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Bootstrap starting.
I20260927 15:21:02.860481 13897 tablet_bootstrap.cc:654] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:02.861902 13897 log.cc:824] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:02.864044 13899 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Bootstrap starting.
I20260927 15:21:02.865175 13897 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: No bootstrap required, opened a new log
I20260927 15:21:02.865456 13897 ts_tablet_manager.cc:1397] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 15:21:02.866315 13900 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Bootstrap starting.
I20260927 15:21:02.868608 13899 tablet_bootstrap.cc:654] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:02.870095 13899 log.cc:824] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:02.870205 13900 tablet_bootstrap.cc:654] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:02.871703 13900 log.cc:824] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:02.876133 13897 raft_consensus.cc:348] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.876551 13897 raft_consensus.cc:374] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:02.876622 13899 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: No bootstrap required, opened a new log
I20260927 15:21:02.876681 13900 tablet_bootstrap.cc:492] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: No bootstrap required, opened a new log
I20260927 15:21:02.876638 13897 raft_consensus.cc:729] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566fbeae24de4c28ac3bee1285fd8931, State: Initialized, Role: FOLLOWER
I20260927 15:21:02.877017 13899 ts_tablet_manager.cc:1397] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260927 15:21:02.877017 13900 ts_tablet_manager.cc:1397] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260927 15:21:02.877336 13897 consensus_queue.cc:260] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.878506 13878 heartbeater.cc:503] Master 127.2.246.126:39533 was elected leader, sending a full tablet report...
I20260927 15:21:02.879233 13897 ts_tablet_manager.cc:1428] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 15:21:02.889187 13899 raft_consensus.cc:348] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.889187 13900 raft_consensus.cc:348] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.889802 13899 raft_consensus.cc:374] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:02.889882 13900 raft_consensus.cc:374] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:02.889959 13899 raft_consensus.cc:729] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a85350d1710447b4b6a0cc857e3ff3eb, State: Initialized, Role: FOLLOWER
I20260927 15:21:02.890002 13900 raft_consensus.cc:729] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54acb8cca8294d51a2ea64d31b8e69b0, State: Initialized, Role: FOLLOWER
I20260927 15:21:02.890823 13900 consensus_queue.cc:260] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.890812 13899 consensus_queue.cc:260] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.892277 13744 heartbeater.cc:503] Master 127.2.246.126:39533 was elected leader, sending a full tablet report...
I20260927 15:21:02.893198 13899 ts_tablet_manager.cc:1428] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260927 15:21:02.893193 13900 ts_tablet_manager.cc:1428] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
W20260927 15:21:02.913362 13745 tablet.cc:2369] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:02.955157 13905 raft_consensus.cc:482] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:21:02.955461 13905 raft_consensus.cc:504] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.957224 13905 leader_election.cc:288] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 566fbeae24de4c28ac3bee1285fd8931 (127.2.246.67:44993), a85350d1710447b4b6a0cc857e3ff3eb (127.2.246.65:41037)
I20260927 15:21:02.967716 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.958008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:40455 (local address 127.2.246.65:41037)
0927 15:21:02.958316 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.958326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.959408 (+ 1082us) server_negotiation.cc:407] Connection header received
0927 15:21:02.959805 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.959819 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.959926 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.960046 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:02.960714 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.961510 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.964806 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.966336 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.966555 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.966650 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.967235 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.967402 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:21:02.967718 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.958019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:47105 (local address 127.2.246.67:44993)
0927 15:21:02.958285 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.958295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.959284 (+ 989us) server_negotiation.cc:407] Connection header received
0927 15:21:02.959492 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.959501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.959598 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.959730 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:02.960450 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.961467 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.964830 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.966330 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.966566 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.966650 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.967235 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.967398 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:21:02.967948 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.957675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:44993 (local address 127.2.246.66:47105)
0927 15:21:02.959097 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.959114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.959329 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.959804 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.959809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.959836 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:02.960293 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.960310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.961621 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.961629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.964545 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.964560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.966691 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.966698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.966951 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.967040 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.967150 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.967349 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":275,"thread_start_us":965,"threads_started":1}
I20260927 15:21:02.968135 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.957701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:41037 (local address 127.2.246.66:40455)
0927 15:21:02.959265 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:02.959287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:02.959501 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:02.960109 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:02.960117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:02.960147 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:02.960557 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.960567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.961698 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.961708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.964544 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:02.964560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.966691 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.966698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:02.966956 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.967065 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 15:21:02.967150 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:21:02.967349 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":361,"thread_start_us":1022,"threads_started":1}
I20260927 15:21:02.969118 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2866f2ecc6c47eb8e647a0d6cc6adec" candidate_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566fbeae24de4c28ac3bee1285fd8931" is_pre_election: true
I20260927 15:21:02.969234 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2866f2ecc6c47eb8e647a0d6cc6adec" candidate_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" is_pre_election: true
I20260927 15:21:02.969684 13833 raft_consensus.cc:2454] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54acb8cca8294d51a2ea64d31b8e69b0 in term 0.
I20260927 15:21:02.969950 13565 raft_consensus.cc:2454] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54acb8cca8294d51a2ea64d31b8e69b0 in term 0.
I20260927 15:21:02.970420 13635 leader_election.cc:302] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54acb8cca8294d51a2ea64d31b8e69b0, 566fbeae24de4c28ac3bee1285fd8931; no voters:
I20260927 15:21:02.970957 13905 raft_consensus.cc:2781] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:21:02.971124 13905 raft_consensus.cc:482] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:21:02.971251 13905 raft_consensus.cc:3037] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:02.973752 13905 raft_consensus.cc:504] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
W20260927 15:21:02.974215 13879 tablet.cc:2369] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:02.974613 13905 leader_election.cc:288] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [CANDIDATE]: Term 1 election: Requested vote from peers 566fbeae24de4c28ac3bee1285fd8931 (127.2.246.67:44993), a85350d1710447b4b6a0cc857e3ff3eb (127.2.246.65:41037)
I20260927 15:21:02.975186 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2866f2ecc6c47eb8e647a0d6cc6adec" candidate_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566fbeae24de4c28ac3bee1285fd8931"
I20260927 15:21:02.975293 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2866f2ecc6c47eb8e647a0d6cc6adec" candidate_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a85350d1710447b4b6a0cc857e3ff3eb"
I20260927 15:21:02.975409 13833 raft_consensus.cc:3037] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:02.975581 13565 raft_consensus.cc:3037] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:02.977717 13833 raft_consensus.cc:2454] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54acb8cca8294d51a2ea64d31b8e69b0 in term 1.
I20260927 15:21:02.978101 13565 raft_consensus.cc:2454] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54acb8cca8294d51a2ea64d31b8e69b0 in term 1.
I20260927 15:21:02.978284 13635 leader_election.cc:302] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54acb8cca8294d51a2ea64d31b8e69b0, 566fbeae24de4c28ac3bee1285fd8931; no voters:
I20260927 15:21:02.978647 13905 raft_consensus.cc:2781] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:02.979226 13905 raft_consensus.cc:686] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 1 LEADER]: Becoming Leader. State: Replica: 54acb8cca8294d51a2ea64d31b8e69b0, State: Running, Role: LEADER
I20260927 15:21:02.979818 13905 consensus_queue.cc:237] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } }
I20260927 15:21:02.985522 13421 catalog_manager.cc:5495] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54acb8cca8294d51a2ea64d31b8e69b0 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54acb8cca8294d51a2ea64d31b8e69b0" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40995 } health_report { overall_health: HEALTHY } } }
I20260927 15:21:03.014921 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.008545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.2.246.66:40995)
0927 15:21:03.008885 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.008891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.008915 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:21:03.009042 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.009046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.009165 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.009357 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:03.009818 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.010671 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.012106 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.012619 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.012767 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:03.013382 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:03.014143 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:03.014278 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.014421 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.014519 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:21:03.025701 13833 raft_consensus.cc:1261] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 1 FOLLOWER]: Refusing update from remote peer 54acb8cca8294d51a2ea64d31b8e69b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:03.025775 13565 raft_consensus.cc:1261] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 1 FOLLOWER]: Refusing update from remote peer 54acb8cca8294d51a2ea64d31b8e69b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:03.026968 13908 consensus_queue.cc:1035] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "566fbeae24de4c28ac3bee1285fd8931" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:03.027618 13905 consensus_queue.cc:1035] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a85350d1710447b4b6a0cc857e3ff3eb" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:03.046377 13911 mvcc.cc:205] Tried to move back new op lower bound from 7333980008540315648 to 7333980008369664000. Current Snapshot: MvccSnapshot[applied={T|T < 7333980008540315648}]
I20260927 15:21:03.048461 13913 mvcc.cc:205] Tried to move back new op lower bound from 7333980008540315648 to 7333980008369664000. Current Snapshot: MvccSnapshot[applied={T|T < 7333980008540315648}]
I20260927 15:21:03.063382 13420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:21:03.063817 13420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:21:03.068492 13420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b844f3f1c69741a9b07f54bbe2b94f79: Sending DeleteTablet for 3 replicas of tablet c2866f2ecc6c47eb8e647a0d6cc6adec
I20260927 15:21:03.069943 13813 tablet_service.cc:1514] Processing DeleteTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:03 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:51968
I20260927 15:21:03.070257 13679 tablet_service.cc:1514] Processing DeleteTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:03 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:59516
I20260927 15:21:03.071143 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet c2866f2ecc6c47eb8e647a0d6cc6adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:03 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260927 15:21:03.071511 13922 tablet_replica.cc:330] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: stopping tablet replica
I20260927 15:21:03.072177 13922 raft_consensus.cc:2229] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:21:03.073100 13922 raft_consensus.cc:2258] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:03.073813 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13482
I20260927 15:21:03.076220 13922 ts_tablet_manager.cc:1905] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:21:03.087178 13922 ts_tablet_manager.cc:1918] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:03.087507 13922 log.cc:1196] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/c2866f2ecc6c47eb8e647a0d6cc6adec
I20260927 15:21:03.088328 13922 ts_tablet_manager.cc:1939] T c2866f2ecc6c47eb8e647a0d6cc6adec P 54acb8cca8294d51a2ea64d31b8e69b0: Deleting consensus metadata
I20260927 15:21:03.090938 13408 catalog_manager.cc:4841] TS 54acb8cca8294d51a2ea64d31b8e69b0 (127.2.246.66:40995): tablet c2866f2ecc6c47eb8e647a0d6cc6adec (table test-table [id=822e2913999945d3ab8923903ef10c53]) successfully deleted
I20260927 15:21:03.095605 13923 tablet_replica.cc:330] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: stopping tablet replica
I20260927 15:21:03.096222 13923 raft_consensus.cc:2229] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:21:03.096933 13923 raft_consensus.cc:2258] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:03.097798 13921 tablet_replica.cc:330] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: stopping tablet replica
I20260927 15:21:03.098332 13921 raft_consensus.cc:2229] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:21:03.098778 13921 raft_consensus.cc:2258] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:03.100168 13923 ts_tablet_manager.cc:1905] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:21:03.100721 13921 ts_tablet_manager.cc:1905] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:21:03.107681 13923 ts_tablet_manager.cc:1918] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:03.107681 13921 ts_tablet_manager.cc:1918] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:03.107848 13921 log.cc:1196] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/c2866f2ecc6c47eb8e647a0d6cc6adec
I20260927 15:21:03.108018 13923 log.cc:1196] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/c2866f2ecc6c47eb8e647a0d6cc6adec
I20260927 15:21:03.108421 13921 ts_tablet_manager.cc:1939] T c2866f2ecc6c47eb8e647a0d6cc6adec P 566fbeae24de4c28ac3bee1285fd8931: Deleting consensus metadata
I20260927 15:21:03.108732 13923 ts_tablet_manager.cc:1939] T c2866f2ecc6c47eb8e647a0d6cc6adec P a85350d1710447b4b6a0cc857e3ff3eb: Deleting consensus metadata
I20260927 15:21:03.110440 13405 catalog_manager.cc:4841] TS a85350d1710447b4b6a0cc857e3ff3eb (127.2.246.65:41037): tablet c2866f2ecc6c47eb8e647a0d6cc6adec (table test-table [id=822e2913999945d3ab8923903ef10c53]) successfully deleted
I20260927 15:21:03.110984 13408 catalog_manager.cc:4841] TS 566fbeae24de4c28ac3bee1285fd8931 (127.2.246.67:44993): tablet c2866f2ecc6c47eb8e647a0d6cc6adec (table test-table [id=822e2913999945d3ab8923903ef10c53]) successfully deleted
I20260927 15:21:03.117657 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.075589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.2.246.65:41037)
0927 15:21:03.075800 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.075807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.094651 (+ 18844us) server_negotiation.cc:407] Connection header received
0927 15:21:03.094921 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.094930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.095061 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.095255 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:03.097638 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.099015 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.101028 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.101789 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.101942 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:03.108135 (+ 6193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:03.108160 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:03.108193 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:03.108272 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:03.113321 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.114416 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.114430 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.114435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.114663 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.115158 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.115164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.115167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.115734 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:03.116094 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.116609 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.116956 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:21:03.118806 13605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:03.229543 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13482
I20260927 15:21:03.259431 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13616
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:21:03.278898 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.260596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.2.246.66:40995)
0927 15:21:03.260789 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.260796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.261256 (+ 460us) server_negotiation.cc:407] Connection header received
0927 15:21:03.261318 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.261326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.261398 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.261515 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:03.262911 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.263785 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.265048 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.265520 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.265690 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:03.270839 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:03.270856 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:03.270873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:03.270919 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:03.274506 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.275885 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.275894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.275897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.276100 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.276848 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.276853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.276855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.277316 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:03.277655 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.278256 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.278455 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:21:03.279961 13739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:03.402348 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13616
I20260927 15:21:03.422032 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13750
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:21:03.439590 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.423065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.2.246.67:44993)
0927 15:21:03.423315 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.423320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.423341 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:21:03.423467 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.423472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.423576 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.423679 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:03.425098 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.426117 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.427483 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:03.427900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:03.428034 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:03.432807 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:03.432830 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:03.432852 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:03.432902 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:03.436274 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.437280 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.437289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.437292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.437447 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:03.437869 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:03.437876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:03.437879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:03.438298 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:03.438633 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.438929 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.439180 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:21:03.440380 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:03.537031 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13750
I20260927 15:21:03.560544 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 13389
I20260927 15:21:03.562173 13451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:03.680826 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 13389
2026-09-27T15:21:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4876 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13931](info): set up 2 sockets
Sep 27 15:21:03 dist-test-slave-14jq krb5kdc[13931](info): commencing operation
krb5kdc: starting...
W20260927 15:21:04.826742 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:21:05 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangeradmin_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangeradmin_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangerlookup_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangerlookup_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
I20260927 15:21:05.222896 3033 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbUGd5v/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/postgres -l logfile start
2026-09-27 15:21:07.985 UTC [13961] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:21:07.985 UTC [13961] LOG: listening on IPv4 address "127.2.246.84", port 40081
2026-09-27 15:21:07.989 UTC [13961] LOG: listening on Unix socket "/tmp/.s.PGSQL.40081"
2026-09-27 15:21:08.021 UTC [13964] LOG: database system was shut down at 2026-09-27 15:21:06 UTC
2026-09-27 15:21:08.025 UTC [13961] LOG: database system is ready to accept connections
I20260927 15:21:09.018508 3033 mini_postgres.cc:96] Postgres bound to 40081
2026-09-27 15:21:09.040 UTC [13972] FATAL: database "slave" does not exist
127.2.246.84:40081 - accepting connections
I20260927 15:21:09.041652 3033 mini_ranger.cc:161] Starting Ranger...
I20260927 15:21:09.074859 3033 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:21:09.686699 3033 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:21:09.686847 3033 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin
2026-09-27 15:21:10,006 [I] DB FLAVOR :POSTGRES
2026-09-27 15:21:10,007 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:21:10,007 [I] Checking connection..
2026-09-27 15:21:10,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:21:10,329 [I] Checking connection passed.
2026-09-27 15:21:10,329 [I] --------- Verifying version history table ---------
2026-09-27 15:21:10,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:21:10,609 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:21:10,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:21:10,888 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:21:10,889 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:21:10,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:21:11.157 UTC [14036] WARNING: there is no transaction in progress
2026-09-27 15:21:11,165 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:21:11,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:21:11,417 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:21:11,418 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:21:11,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:21:11,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:21:11,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-14jq.c.gcp-upstream.internal','N') ;"
2026-09-27 15:21:12,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:21:12,592 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:21:12,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:21:12,910 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:21:12,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:21:13,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:21:13,523 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:21:13,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:21:14.713 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.731 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.750 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.768 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.789 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.812 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.890 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.899 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.918 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.932 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.945 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.965 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.978 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.988 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:14.998 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:15.010 UTC [14144] WARNING: there is no transaction in progress
2026-09-27 15:21:15,932 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:21:15,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:21:16,195 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:21:16,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:21:16,469 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:21:16,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:21:16,784 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:21:16,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:21:17,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:21:17,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-14jq.c.gcp-upstream.internal';"
2026-09-27 15:21:17,597 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:21:17,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:21:17,878 [I] DB_PATCHES have already been applied
I20260927 15:21:17.885927 3033 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbUGd5v/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:21:17.886049 3033 mini_ranger.cc:193] Using host: 127.2.246.84
I20260927 15:21:17.905205 3033 mini_ranger.cc:227] Ranger admin URL: http://127.2.246.84:55895
Sep 27, 2026 3:21:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:21:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:21:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:21:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:21:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:21:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:21:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:21:19.126209 3033 mini_ranger.cc:160] Time spent starting Ranger: real 10.085s user 0.007s sys 0.049s
Sep 27, 2026 3:21:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:21:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 3:21:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:21:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:21:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:21:44.931968 3033 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangerkms_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/rangerkms_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:21:45 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:21:45.154848 3033 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:21:45.185946 3033 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:21:46.190341 3033 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:21:46.190507 3033 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms
I20260927 15:21:46.190725 3033 mini_ranger_kms.cc:227] Using postgres at 127.2.246.84:40081
2026-09-27 15:21:46,628 [I] DB FLAVOR :POSTGRES
2026-09-27 15:21:46,629 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:21:46,629 [I] Checking connection
2026-09-27 15:21:46,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:21:46,916 [I] connection success
2026-09-27 15:21:46,916 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:21:46,916 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:21:46,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:21:47,196 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:21:47,197 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:21:47,197 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:21:47,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:40081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:21:47,500 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:21:48.059652 3033 mini_ranger_kms.cc:310] Created kms service
I20260927 15:21:48.262148 3033 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:21:48.318607 3033 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:21:48 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:48.857791 3033 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:21:48.858247 3033 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:21:48.858332 3033 mini_ranger_kms.cc:240] Using host: 127.2.246.84
I20260927 15:21:48.884259 3033 mini_ranger_kms.cc:276] Ranger KMS PID: 14325
I20260927 15:21:48.884568 3033 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.246.84:34699
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:21:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:21:51.274638 3033 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.120s user 0.129s sys 0.334s
I20260927 15:21:51.274890 3033 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:21:51.275116 3033 mini_ranger_kms.cc:401] 127.2.246.84:34699/kms/v1/keys
Sep 27 15:21:51 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:21:51 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
Sep 27 15:21:52 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
Sep 27 15:21:54 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:21:54.918740 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39519
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:39519
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:34699/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:55.383983 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:55.384488 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:55.384675 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:55.393354 14375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:55.393572 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:55.393700 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:55.393810 14375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:55.393935 14375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:55.407218 14375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.246.84:34699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:39519
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:55.409283 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:55.411203 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:55.427449 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:55.427356 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:55.427449 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:55.428788 14375 server_base.cc:1034] running on GCE node
I20260927 15:21:55.429919 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:55.432783 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:55.434036 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790522515433953 us; error 68 us; skew 500 ppm
I20260927 15:21:55.434554 14375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:55 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:55.438985 14375 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:21:55.441056 14375 webserver.cc:460] Webserver started at http://127.2.246.126:41447/ using document root <none> and password file <none>
I20260927 15:21:55.442054 14375 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:55.442227 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:55.442579 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:21:55 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:55.505869 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "0dc6cb3e71c147828e2e24de1e7cefed"
format_stamp: "Formatted at 2026-09-27 15:21:55 on dist-test-slave-14jq"
server_key: "15a711232257e8f20ed02fb3c2068158"
server_key_iv: "8e89e678de5c3a5d03d3c66757b6d7cf"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:55.506983 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "0dc6cb3e71c147828e2e24de1e7cefed"
format_stamp: "Formatted at 2026-09-27 15:21:55 on dist-test-slave-14jq"
server_key: "15a711232257e8f20ed02fb3c2068158"
server_key_iv: "8e89e678de5c3a5d03d3c66757b6d7cf"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:55.514415 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 15:21:55.539518 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:55.541656 14375 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 15:21:55.541997 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "0dc6cb3e71c147828e2e24de1e7cefed"
format_stamp: "Formatted at 2026-09-27 15:21:55 on dist-test-slave-14jq"
server_key: "15a711232257e8f20ed02fb3c2068158"
server_key_iv: "8e89e678de5c3a5d03d3c66757b6d7cf"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:55.542300 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:55.570461 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:55.575975 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:55.576380 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:55.602450 14375 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:39519
I20260927 15:21:55.602466 14441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:39519 every 8 connection(s)
I20260927 15:21:55.604503 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:55.606321 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 14375
I20260927 15:21:55.606562 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:21:55.607265 3033 external_mini_cluster.cc:1428] Setting key 3f8d3b09087dc2d824fa0599e82cab72
I20260927 15:21:55.611080 14442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:55.628213 14442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: Bootstrap starting.
I20260927 15:21:55.632366 14442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:55.633522 14442 log.cc:824] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:55.637957 14442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: No bootstrap required, opened a new log
Sep 27 15:21:55 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:21:55.649020 14442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } }
I20260927 15:21:55.649520 14442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:55.649652 14442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc6cb3e71c147828e2e24de1e7cefed, State: Initialized, Role: FOLLOWER
I20260927 15:21:55.650413 14442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } }
I20260927 15:21:55.650743 14442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:21:55.650907 14442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:21:55.651100 14442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:55.651413 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:55.610152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.2.246.126:39519)
0927 15:21:55.624630 (+ 14478us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:55.624649 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:55.624718 (+ 69us) server_negotiation.cc:407] Connection header received
0927 15:21:55.627074 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:55.627109 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:55.627715 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:55.628260 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:55.630046 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:55.632069 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:55.633793 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:55.635472 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:55.635706 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:55.640429 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:55.640441 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:55.640464 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:55.640521 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:55.645310 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:55.646524 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:55.646539 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:55.646549 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:55.646780 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:55.647330 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:55.647338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:55.647341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:55.648216 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:55.648522 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:55.649165 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 15:21:55.649814 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14087,"thread_start_us":15597,"threads_started":1}
I20260927 15:21:55.654639 14442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } }
I20260927 15:21:55.655228 14442 leader_election.cc:302] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dc6cb3e71c147828e2e24de1e7cefed; no voters:
I20260927 15:21:55.656010 14442 leader_election.cc:288] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:21:55.656219 14447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:55.657328 14447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [term 1 LEADER]: Becoming Leader. State: Replica: 0dc6cb3e71c147828e2e24de1e7cefed, State: Running, Role: LEADER
I20260927 15:21:55.658107 14447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } }
I20260927 15:21:55.658465 14442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:55.662187 14449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dc6cb3e71c147828e2e24de1e7cefed. Latest consensus state: current_term: 1 leader_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } } }
I20260927 15:21:55.662690 14449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:55.663326 14448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc6cb3e71c147828e2e24de1e7cefed" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39519 } } }
I20260927 15:21:55.663803 14448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:55.666478 14453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:55.675313 14453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:55.677001 14442 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:21:55.698686 14453 catalog_manager.cc:1347] Generated new cluster ID: d613121ac4af4c75bcb94ca3a2454874
I20260927 15:21:55.698841 14453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:55.739979 14453 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:55.742663 14453 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:55.755921 14453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: Generated new TSK 0
I20260927 15:21:55.757278 14453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:21:56.540597 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:34699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:21:56 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:21:57.023042 14487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:57.023561 14487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:57.023774 14487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:57.036044 14487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:57.036221 14487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:57.036496 14487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:21:57.049938 14487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.246.84:34699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:57.051427 14487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:57.052798 14487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:57.070364 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:57.071197 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:57.071660 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:57.078045 14487 server_base.cc:1034] running on GCE node
I20260927 15:21:57.079016 14487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:57.080461 14487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:57.082005 14487 hybrid_clock.cc:648] HybridClock initialized: now 1790522517081924 us; error 86 us; skew 500 ppm
I20260927 15:21:57.082476 14487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:57 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:57.087054 14487 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:21:57.103511 14487 webserver.cc:460] Webserver started at http://127.2.246.65:42521/ using document root <none> and password file <none>
I20260927 15:21:57.104374 14487 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:57.104540 14487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:57.104897 14487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:21:57 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:57.122582 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "47cf1c67796f4da39fccd00cb2a77016"
format_stamp: "Formatted at 2026-09-27 15:21:57 on dist-test-slave-14jq"
server_key: "11326ede3f96773558ce1094fa560ee2"
server_key_iv: "efb89a26c53f7d9deb029e54587a0d16"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:57.123867 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "47cf1c67796f4da39fccd00cb2a77016"
format_stamp: "Formatted at 2026-09-27 15:21:57 on dist-test-slave-14jq"
server_key: "11326ede3f96773558ce1094fa560ee2"
server_key_iv: "efb89a26c53f7d9deb029e54587a0d16"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:57.144942 14487 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.016s sys 0.003s
I20260927 15:21:57.163467 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:57.165133 14487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:21:57.165737 14487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "47cf1c67796f4da39fccd00cb2a77016"
format_stamp: "Formatted at 2026-09-27 15:21:57 on dist-test-slave-14jq"
server_key: "11326ede3f96773558ce1094fa560ee2"
server_key_iv: "efb89a26c53f7d9deb029e54587a0d16"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:57.166042 14487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:57.218487 14487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:57.223788 14487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:57.224185 14487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:57.227823 14487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:57.237365 14487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:57.237566 14487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:21:57.237740 14487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:57.237814 14487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:57.323542 14487 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:42727
I20260927 15:21:57.323628 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:42727 every 8 connection(s)
I20260927 15:21:57.325762 14487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:21:57.331815 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 14487
I20260927 15:21:57.332034 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:21:57.333007 3033 external_mini_cluster.cc:1428] Setting key 3b1844f415bc5d1f72e43abed07c24c8
Sep 27 15:21:57 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:21:57.391803 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:57.332001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:38653 (local address 127.2.246.126:39519)
0927 15:21:57.346930 (+ 14929us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:57.346938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:57.349173 (+ 2235us) server_negotiation.cc:407] Connection header received
0927 15:21:57.351103 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:57.351111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:57.351229 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:57.351399 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:57.358741 (+ 7342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:57.359977 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:57.363528 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:57.364256 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:57.364375 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:57.381540 (+ 17165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:57.381553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:57.381561 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:57.381653 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:57.386468 (+ 4815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:57.388193 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:57.388200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:57.388205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:57.388446 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:57.389166 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:57.389171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:57.389176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:57.389615 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:57.389770 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:57.391012 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0927 15:21:57.391329 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14738,"thread_start_us":14474,"threads_started":1}
I20260927 15:21:57.395543 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:57.330160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39519 (local address 127.2.246.65:38653)
0927 15:21:57.347811 (+ 17651us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:57.347905 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:57.350656 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:57.351781 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:57.351799 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:57.353694 (+ 1895us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:57.358358 (+ 4664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:57.358402 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:57.360527 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:57.360535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:57.363204 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:57.363243 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:57.364846 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:57.364856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:57.365249 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:57.374217 (+ 8968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:57.374269 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:57.381041 (+ 6772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:57.386833 (+ 5792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:57.386848 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:57.386877 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:57.387948 (+ 1071us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:57.388615 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:57.388620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:57.388624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:57.388999 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:57.389916 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:57.389927 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:57.390521 (+ 594us) client_negotiation.cc:769] Sending connection context
0927 15:21:57.392546 (+ 2025us) client_negotiation.cc:240] Negotiation successful
0927 15:21:57.393314 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17131,"thread_start_us":16854,"threads_started":1}
I20260927 15:21:57.400950 14620 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39519
I20260927 15:21:57.401522 14620 heartbeater.cc:464] Registering TS with master...
Sep 27 15:21:57 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:57.402917 14620 heartbeater.cc:511] Master 127.2.246.126:39519 requested a full tablet report, sending...
I20260927 15:21:57.407182 14407 ts_manager.cc:194] Registered new tserver with Master: 47cf1c67796f4da39fccd00cb2a77016 (127.2.246.65:42727)
I20260927 15:21:57.413378 14407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:38653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
W20260927 15:21:57.442495 3033 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.246.66: real 0.106s user 0.002s sys 0.016s
I20260927 15:21:57.443641 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:34699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:21:57 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
W20260927 15:21:57.960140 14626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:57.960673 14626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:57.960870 14626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:57.973397 14626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:57.973670 14626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:57.973939 14626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:21:57.992015 14626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.246.84:34699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:57.994022 14626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:57.995843 14626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:58.025563 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:58.026196 14633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:58.032732 14626 server_base.cc:1034] running on GCE node
W20260927 15:21:58.033871 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:58.035082 14626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:58.055251 14626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:58.056766 14626 hybrid_clock.cc:648] HybridClock initialized: now 1790522518056637 us; error 146 us; skew 500 ppm
I20260927 15:21:58.057286 14626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:58.063462 14626 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:21:58.067149 14626 webserver.cc:460] Webserver started at http://127.2.246.66:35579/ using document root <none> and password file <none>
I20260927 15:21:58.069027 14626 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:58.069685 14626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:58.070569 14626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:58.100741 14626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "ee1cb04d83fb4487b4e7ecc749380502"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "97508c95822b3d00424a4c802295d40e"
server_key_iv: "831db233f6579fece75c2574827d77c5"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.102223 14626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "ee1cb04d83fb4487b4e7ecc749380502"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "97508c95822b3d00424a4c802295d40e"
server_key_iv: "831db233f6579fece75c2574827d77c5"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.125049 14626 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.009s sys 0.011s
I20260927 15:21:58.163653 14650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.166435 14626 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.004s
I20260927 15:21:58.166759 14626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "ee1cb04d83fb4487b4e7ecc749380502"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "97508c95822b3d00424a4c802295d40e"
server_key_iv: "831db233f6579fece75c2574827d77c5"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.167088 14626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:58.203761 14626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:58.208730 14626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:58.209583 14626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:58.211033 14626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:58.213229 14626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:58.213347 14626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.213482 14626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:58.213546 14626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.264102 14626 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:34509
I20260927 15:21:58.264122 14762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:34509 every 8 connection(s)
I20260927 15:21:58.265532 14626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:21:58.272279 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 14626
I20260927 15:21:58.272491 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:21:58.273118 3033 external_mini_cluster.cc:1428] Setting key bd7aa6bfa801172a686066aa08bffe24
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:21:58.305120 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:58.270041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:37967 (local address 127.2.246.126:39519)
0927 15:21:58.270720 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:58.270727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:58.272685 (+ 1958us) server_negotiation.cc:407] Connection header received
0927 15:21:58.280976 (+ 8291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:58.280983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:58.281105 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:58.281198 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:58.284022 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.285264 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.287332 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.287823 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.287990 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:58.294756 (+ 6766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:58.294767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:58.294773 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:58.294847 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:58.299054 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.300474 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.300481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.300486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.300710 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.301827 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.301833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.301990 (+ 157us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.302927 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:58.303136 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:58.304328 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0927 15:21:58.304673 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260927 15:21:58.305807 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:58.271182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39519 (local address 127.2.246.66:37967)
0927 15:21:58.272453 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:58.272529 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:58.280495 (+ 7966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:58.281993 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:58.282008 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:58.282753 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:58.283760 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:58.283790 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.285518 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.285523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:58.287131 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:58.287161 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.288233 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.288246 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:58.288732 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:58.290305 (+ 1573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:58.290352 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:58.294239 (+ 3887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:58.299353 (+ 5114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:58.299367 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:58.299390 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:58.300213 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:58.300872 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:58.300880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:58.300886 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:58.301634 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:58.303134 (+ 1500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:58.303146 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:58.303711 (+ 565us) client_negotiation.cc:769] Sending connection context
0927 15:21:58.304101 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 15:21:58.304566 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":2624,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:21:58.308095 14763 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39519
I20260927 15:21:58.308578 14763 heartbeater.cc:464] Registering TS with master...
I20260927 15:21:58.309689 14763 heartbeater.cc:511] Master 127.2.246.126:39519 requested a full tablet report, sending...
I20260927 15:21:58.312075 14407 ts_manager.cc:194] Registered new tserver with Master: ee1cb04d83fb4487b4e7ecc749380502 (127.2.246.66:34509)
I20260927 15:21:58.313409 14407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:37967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:21:58.339427 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:34699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:21:58.419000 14620 heartbeater.cc:503] Master 127.2.246.126:39519 was elected leader, sending a full tablet report...
W20260927 15:21:58.707044 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:58.707399 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:58.707566 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:58.716537 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:58.716727 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:58.716907 14769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:21:58.727778 14769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.246.84:34699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:21:58.729965 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:58.732252 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:58.747826 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:58.748113 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:58.748476 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:58.750305 14769 server_base.cc:1034] running on GCE node
I20260927 15:21:58.751122 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:58.752321 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:58.753571 14769 hybrid_clock.cc:648] HybridClock initialized: now 1790522518753484 us; error 70 us; skew 500 ppm
I20260927 15:21:58.754204 14769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:58.758337 14769 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:21:58.760605 14769 webserver.cc:460] Webserver started at http://127.2.246.67:41539/ using document root <none> and password file <none>
I20260927 15:21:58.761566 14769 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:58.761752 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:58.762203 14769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:21:58.784194 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "514bead995a94caea03907d9d9c5ee93"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "a60ff79c455f68bbe0aac90c6febabf1"
server_key_iv: "1e27d31c1f4b031964204e71135173ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.785076 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "514bead995a94caea03907d9d9c5ee93"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "a60ff79c455f68bbe0aac90c6febabf1"
server_key_iv: "1e27d31c1f4b031964204e71135173ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.791539 14769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:21:58.813541 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.815925 14769 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 15:21:58.816167 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "514bead995a94caea03907d9d9c5ee93"
format_stamp: "Formatted at 2026-09-27 15:21:58 on dist-test-slave-14jq"
server_key: "a60ff79c455f68bbe0aac90c6febabf1"
server_key_iv: "1e27d31c1f4b031964204e71135173ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:21:58.816408 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:58.865595 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:58.870604 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:58.870942 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:58.872104 14769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:21:58.873729 14769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:21:58.873811 14769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.873966 14769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:21:58.874018 14769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:58.916402 14769 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:41903
I20260927 15:21:58.916422 14896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:41903 every 8 connection(s)
I20260927 15:21:58.917770 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:21:58.926923 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 14769
I20260927 15:21:58.927169 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:21:58.927785 3033 external_mini_cluster.cc:1428] Setting key 8c25ddb66f754291ca80e32645c181db
Sep 27 15:21:58 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:21:58.944617 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:58.922052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:51543 (local address 127.2.246.126:39519)
0927 15:21:58.922781 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:58.922787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:58.923086 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:21:58.925162 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:58.925167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:58.925246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:58.925378 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:58.927997 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.929086 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.931757 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.932430 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.932569 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:58.937225 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:58.937234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:58.937239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:58.937284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:58.940321 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.941326 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.941333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.941337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.941478 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.942045 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.942050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.942053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.942372 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:58.942542 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:58.943432 (+ 890us) server_negotiation.cc:299] Negotiation successful
0927 15:21:58.944175 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":379,"threads_started":1}
I20260927 15:21:58.945272 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:58.921647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39519 (local address 127.2.246.67:51543)
0927 15:21:58.922858 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:58.922916 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:58.924812 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:58.925721 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:58.925732 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:58.926482 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:58.927628 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:58.927665 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.929279 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.929284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:58.931505 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:58.931525 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.932743 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.932749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:58.933047 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:58.934028 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:58.934058 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:58.936806 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:58.940547 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:58.940566 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:58.940592 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:58.941171 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:58.941616 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:58.941622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:58.941625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:58.941909 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:58.942517 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:58.942525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:58.942998 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 15:21:58.943326 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 15:21:58.943875 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":630,"threads_started":1}
I20260927 15:21:58.947027 14897 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39519
I20260927 15:21:58.948359 14897 heartbeater.cc:464] Registering TS with master...
I20260927 15:21:58.949369 14897 heartbeater.cc:511] Master 127.2.246.126:39519 requested a full tablet report, sending...
I20260927 15:21:58.951332 14407 ts_manager.cc:194] Registered new tserver with Master: 514bead995a94caea03907d9d9c5ee93 (127.2.246.67:41903)
I20260927 15:21:58.952232 14407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:51543
I20260927 15:21:58.958915 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:21:58.980198 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:58.964803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.2.246.126:39519)
0927 15:21:58.965122 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:58.965133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:58.965628 (+ 495us) server_negotiation.cc:407] Connection header received
0927 15:21:58.965884 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:58.965892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:58.965987 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:58.966099 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:58.967963 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.968780 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.970597 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:58.971152 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:58.971300 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:58.974245 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:58.974250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:58.974254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:58.974295 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:58.977027 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.978188 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.978192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.978194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.978328 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:58.978751 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:58.978755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:58.978757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:58.979031 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:58.979232 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:58.979623 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 15:21:58.979824 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:21:58.986191 14407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:21:58.995883 14407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:21:59.031352 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.015093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:34509 (local address 127.0.0.1:43436)
0927 15:21:59.016703 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:59.016748 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:59.016970 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:59.018931 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:59.018946 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:59.019004 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:59.019764 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.019785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.022844 (+ 3059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.022856 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.027035 (+ 4179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.027081 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.029883 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.029892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.030268 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.030339 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:21:59.030651 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 15:21:59.030889 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":825,"threads_started":1}
I20260927 15:21:59.031338 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.015833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.2.246.66:34509)
0927 15:21:59.017315 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.017332 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.017385 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:21:59.017595 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.017609 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.018096 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.018443 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:59.020091 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.022532 (+ 2441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.027481 (+ 4949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.029462 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.029636 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.029931 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.030729 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.031002 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":1120,"threads_started":1}
I20260927 15:21:59.037140 14698 tablet_service.cc:1467] Processing CreateTablet for tablet c0f5e51ee7b743de802b5162e2a19417 (DEFAULT_TABLE table=test-table [id=cf67fed8ff454625beb28a650d6b362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:59.039566 14698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f5e51ee7b743de802b5162e2a19417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:59.040159 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.016501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:41903 (local address 127.0.0.1:55110)
0927 15:21:59.027787 (+ 11286us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:59.027820 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:59.028009 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:59.030771 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:59.030780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:59.030811 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:59.031292 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.031306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.034385 (+ 3079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.034390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.036402 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.036420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.038994 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.039002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.039228 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.039270 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:21:59.039395 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:21:59.039634 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11092,"mutex_wait_us":24,"thread_start_us":13667,"threads_started":1}
I20260927 15:21:59.040154 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.017548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.2.246.67:41903)
0927 15:21:59.029453 (+ 11905us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.029465 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.029515 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:21:59.029652 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.029662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.030110 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.030404 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:59.032701 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.034188 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.036659 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.038311 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.038500 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.038800 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.039499 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.039742 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11655,"thread_start_us":11443,"threads_started":1}
I20260927 15:21:59.040246 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.018062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:47962)
0927 15:21:59.030550 (+ 12488us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:59.030576 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:59.030736 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:59.032282 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:59.032292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:59.032331 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:59.032934 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.032948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.035186 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.035192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.037021 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.037031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.039291 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.039298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.039489 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.039526 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:21:59.039619 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:21:59.039780 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12314,"mutex_wait_us":1,"thread_start_us":9329,"threads_started":1}
I20260927 15:21:59.041051 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.017391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.2.246.65:42727)
0927 15:21:59.029884 (+ 12493us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.029895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.030903 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 15:21:59.031060 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.031074 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.031441 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.031695 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:59.033262 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.034983 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.037236 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.038880 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.039027 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.039302 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.040531 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.040742 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12279,"thread_start_us":12072,"threads_started":1}
I20260927 15:21:59.044839 14555 tablet_service.cc:1467] Processing CreateTablet for tablet c0f5e51ee7b743de802b5162e2a19417 (DEFAULT_TABLE table=test-table [id=cf67fed8ff454625beb28a650d6b362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:59.044744 14832 tablet_service.cc:1467] Processing CreateTablet for tablet c0f5e51ee7b743de802b5162e2a19417 (DEFAULT_TABLE table=test-table [id=cf67fed8ff454625beb28a650d6b362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:21:59.046490 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f5e51ee7b743de802b5162e2a19417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:59.046497 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f5e51ee7b743de802b5162e2a19417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:59.057355 14916 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Bootstrap starting.
I20260927 15:21:59.063436 14917 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Bootstrap starting.
I20260927 15:21:59.064131 14916 tablet_bootstrap.cc:654] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:59.064783 14918 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Bootstrap starting.
I20260927 15:21:59.066089 14916 log.cc:824] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:59.070631 14917 tablet_bootstrap.cc:654] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:59.071444 14918 tablet_bootstrap.cc:654] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:59.071532 14916 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: No bootstrap required, opened a new log
I20260927 15:21:59.071914 14916 ts_tablet_manager.cc:1397] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260927 15:21:59.072055 14917 log.cc:824] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:59.073226 14918 log.cc:824] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:59.076560 14917 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: No bootstrap required, opened a new log
I20260927 15:21:59.076959 14917 ts_tablet_manager.cc:1397] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20260927 15:21:59.077229 14918 tablet_bootstrap.cc:492] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: No bootstrap required, opened a new log
I20260927 15:21:59.077538 14918 ts_tablet_manager.cc:1397] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260927 15:21:59.081197 14916 raft_consensus.cc:348] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.081703 14916 raft_consensus.cc:374] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:59.081810 14916 raft_consensus.cc:729] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1cb04d83fb4487b4e7ecc749380502, State: Initialized, Role: FOLLOWER
I20260927 15:21:59.082489 14916 consensus_queue.cc:260] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.084299 14916 ts_tablet_manager.cc:1428] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 15:21:59.084590 14763 heartbeater.cc:503] Master 127.2.246.126:39519 was elected leader, sending a full tablet report...
I20260927 15:21:59.088862 14918 raft_consensus.cc:348] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.089433 14918 raft_consensus.cc:374] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:59.089125 14917 raft_consensus.cc:348] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.089564 14918 raft_consensus.cc:729] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cf1c67796f4da39fccd00cb2a77016, State: Initialized, Role: FOLLOWER
I20260927 15:21:59.089676 14917 raft_consensus.cc:374] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:59.089805 14917 raft_consensus.cc:729] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514bead995a94caea03907d9d9c5ee93, State: Initialized, Role: FOLLOWER
I20260927 15:21:59.090528 14917 consensus_queue.cc:260] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.090654 14918 consensus_queue.cc:260] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.092518 14897 heartbeater.cc:503] Master 127.2.246.126:39519 was elected leader, sending a full tablet report...
I20260927 15:21:59.093073 14917 ts_tablet_manager.cc:1428] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 15:21:59.095323 14918 ts_tablet_manager.cc:1428] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
W20260927 15:21:59.169734 14898 tablet.cc:2369] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:59.268242 14924 raft_consensus.cc:482] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 15:21:59.268923 14764 tablet.cc:2369] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:59.269364 14924 raft_consensus.cc:504] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.272435 14924 leader_election.cc:288] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1cb04d83fb4487b4e7ecc749380502 (127.2.246.66:34509), 514bead995a94caea03907d9d9c5ee93 (127.2.246.67:41903)
I20260927 15:21:59.283177 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.273106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:54123 (local address 127.2.246.67:41903)
0927 15:21:59.273375 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.273383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.274826 (+ 1443us) server_negotiation.cc:407] Connection header received
0927 15:21:59.275044 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.275049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.275141 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.275243 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:59.276299 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.277158 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.280714 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.281872 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.282102 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.282204 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.282620 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.282820 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:21:59.283215 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.272999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:41903 (local address 127.2.246.65:54123)
0927 15:21:59.274672 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:59.274696 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:59.274875 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:59.275302 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:59.275308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:59.275351 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:59.276034 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.276048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.277343 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.277349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.280419 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.280446 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.282173 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.282181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.282386 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.282427 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:21:59.282556 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:21:59.282726 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":46,"thread_start_us":1239,"threads_started":1}
I20260927 15:21:59.284953 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.273521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:54509 (local address 127.2.246.66:34509)
0927 15:21:59.273806 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.273814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.274333 (+ 519us) server_negotiation.cc:407] Connection header received
0927 15:21:59.274728 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.274734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.274861 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.274958 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:21:59.276428 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.277786 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.281914 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.283355 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.283608 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.283720 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.284458 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.284734 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 15:21:59.285082 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.272914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:34509 (local address 127.2.246.65:54509)
0927 15:21:59.274156 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:59.274189 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:59.274511 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:59.275211 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:59.275218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:59.275249 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:21:59.276190 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.276228 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.278030 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.278038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.281551 (+ 3513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:59.281582 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.283949 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.283956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:59.284147 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.284192 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:21:59.284329 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 15:21:59.284522 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":110,"spinlock_wait_cycles":37120,"thread_start_us":920,"threads_started":1}
I20260927 15:21:59.284902 14852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f5e51ee7b743de802b5162e2a19417" candidate_uuid: "47cf1c67796f4da39fccd00cb2a77016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "514bead995a94caea03907d9d9c5ee93" is_pre_election: true
I20260927 15:21:59.285686 14852 raft_consensus.cc:2454] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cf1c67796f4da39fccd00cb2a77016 in term 0.
I20260927 15:21:59.286222 14718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f5e51ee7b743de802b5162e2a19417" candidate_uuid: "47cf1c67796f4da39fccd00cb2a77016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1cb04d83fb4487b4e7ecc749380502" is_pre_election: true
I20260927 15:21:59.286768 14718 raft_consensus.cc:2454] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cf1c67796f4da39fccd00cb2a77016 in term 0.
I20260927 15:21:59.286849 14511 leader_election.cc:302] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf1c67796f4da39fccd00cb2a77016, 514bead995a94caea03907d9d9c5ee93; no voters:
I20260927 15:21:59.287436 14924 raft_consensus.cc:2781] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:21:59.287618 14924 raft_consensus.cc:482] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:21:59.287741 14924 raft_consensus.cc:3037] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:59.290298 14924 raft_consensus.cc:504] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.291141 14924 leader_election.cc:288] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [CANDIDATE]: Term 1 election: Requested vote from peers ee1cb04d83fb4487b4e7ecc749380502 (127.2.246.66:34509), 514bead995a94caea03907d9d9c5ee93 (127.2.246.67:41903)
I20260927 15:21:59.292241 14718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f5e51ee7b743de802b5162e2a19417" candidate_uuid: "47cf1c67796f4da39fccd00cb2a77016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1cb04d83fb4487b4e7ecc749380502"
I20260927 15:21:59.292241 14852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f5e51ee7b743de802b5162e2a19417" candidate_uuid: "47cf1c67796f4da39fccd00cb2a77016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "514bead995a94caea03907d9d9c5ee93"
I20260927 15:21:59.292599 14852 raft_consensus.cc:3037] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:59.292599 14718 raft_consensus.cc:3037] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:59.298166 14852 raft_consensus.cc:2454] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cf1c67796f4da39fccd00cb2a77016 in term 1.
I20260927 15:21:59.298166 14718 raft_consensus.cc:2454] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cf1c67796f4da39fccd00cb2a77016 in term 1.
I20260927 15:21:59.299001 14511 leader_election.cc:302] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf1c67796f4da39fccd00cb2a77016, 514bead995a94caea03907d9d9c5ee93; no voters:
I20260927 15:21:59.299621 14924 raft_consensus.cc:2781] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:59.300325 14924 raft_consensus.cc:686] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 1 LEADER]: Becoming Leader. State: Replica: 47cf1c67796f4da39fccd00cb2a77016, State: Running, Role: LEADER
I20260927 15:21:59.301033 14924 consensus_queue.cc:237] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:21:59.307876 14406 catalog_manager.cc:5495] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cf1c67796f4da39fccd00cb2a77016 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "47cf1c67796f4da39fccd00cb2a77016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47cf1c67796f4da39fccd00cb2a77016" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } health_report { overall_health: HEALTHY } } }
W20260927 15:21:59.336400 14621 tablet.cc:2369] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:59.345508 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.338665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.2.246.65:42727)
0927 15:21:59.338897 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.338902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.339037 (+ 135us) server_negotiation.cc:407] Connection header received
0927 15:21:59.339210 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.339214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.339301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.339480 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:59.340036 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.340849 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.342908 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.343386 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.343516 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.344246 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:59.344872 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:59.344999 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.345190 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.345272 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:21:59.357254 14852 raft_consensus.cc:1261] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 1 FOLLOWER]: Refusing update from remote peer 47cf1c67796f4da39fccd00cb2a77016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:59.357268 14718 raft_consensus.cc:1261] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 1 FOLLOWER]: Refusing update from remote peer 47cf1c67796f4da39fccd00cb2a77016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:59.359090 14927 consensus_queue.cc:1035] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "514bead995a94caea03907d9d9c5ee93" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:59.359779 14924 consensus_queue.cc:1035] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1cb04d83fb4487b4e7ecc749380502" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:59.406215 14407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:21:59.407037 14407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:21:59.413165 14407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dc6cb3e71c147828e2e24de1e7cefed: Sending DeleteTablet for 3 replicas of tablet c0f5e51ee7b743de802b5162e2a19417
I20260927 15:21:59.414842 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f5e51ee7b743de802b5162e2a19417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:59 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:55110
I20260927 15:21:59.414960 14698 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f5e51ee7b743de802b5162e2a19417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:59 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 15:21:59.417339 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f5e51ee7b743de802b5162e2a19417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:21:59 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:47962
I20260927 15:21:59.417573 14940 tablet_replica.cc:330] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: stopping tablet replica
I20260927 15:21:59.418329 14940 raft_consensus.cc:2229] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:21:59.419685 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 14487
I20260927 15:21:59.419723 14940 raft_consensus.cc:2258] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:59.421911 14941 tablet_replica.cc:330] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: stopping tablet replica
I20260927 15:21:59.422677 14941 raft_consensus.cc:2229] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:21:59.423789 14941 raft_consensus.cc:2258] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:59.427466 14940 ts_tablet_manager.cc:1905] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:21:59.432507 14941 ts_tablet_manager.cc:1905] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:21:59.435921 14942 tablet_replica.cc:330] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: stopping tablet replica
I20260927 15:21:59.436517 14942 raft_consensus.cc:2229] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:21:59.437320 14942 raft_consensus.cc:2258] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:21:59.442744 14940 ts_tablet_manager.cc:1918] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:59.443061 14940 log.cc:1196] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/c0f5e51ee7b743de802b5162e2a19417
I20260927 15:21:59.443890 14940 ts_tablet_manager.cc:1939] T c0f5e51ee7b743de802b5162e2a19417 P 514bead995a94caea03907d9d9c5ee93: Deleting consensus metadata
I20260927 15:21:59.446476 14942 ts_tablet_manager.cc:1905] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:21:59.447664 14394 catalog_manager.cc:4841] TS 514bead995a94caea03907d9d9c5ee93 (127.2.246.67:41903): tablet c0f5e51ee7b743de802b5162e2a19417 (table test-table [id=cf67fed8ff454625beb28a650d6b362a]) successfully deleted
I20260927 15:21:59.449332 14941 ts_tablet_manager.cc:1918] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:59.449630 14941 log.cc:1196] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/c0f5e51ee7b743de802b5162e2a19417
I20260927 15:21:59.450533 14941 ts_tablet_manager.cc:1939] T c0f5e51ee7b743de802b5162e2a19417 P ee1cb04d83fb4487b4e7ecc749380502: Deleting consensus metadata
Sep 27 15:21:59 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:21:59.455039 14392 catalog_manager.cc:4841] TS ee1cb04d83fb4487b4e7ecc749380502 (127.2.246.66:34509): tablet c0f5e51ee7b743de802b5162e2a19417 (table test-table [id=cf67fed8ff454625beb28a650d6b362a]) successfully deleted
I20260927 15:21:59.458274 14942 ts_tablet_manager.cc:1918] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:21:59.458552 14942 log.cc:1196] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/c0f5e51ee7b743de802b5162e2a19417
I20260927 15:21:59.459197 14942 ts_tablet_manager.cc:1939] T c0f5e51ee7b743de802b5162e2a19417 P 47cf1c67796f4da39fccd00cb2a77016: Deleting consensus metadata
I20260927 15:21:59.461508 14392 catalog_manager.cc:4841] TS 47cf1c67796f4da39fccd00cb2a77016 (127.2.246.65:42727): tablet c0f5e51ee7b743de802b5162e2a19417 (table test-table [id=cf67fed8ff454625beb28a650d6b362a]) successfully deleted
I20260927 15:21:59.463397 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.422105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.2.246.65:42727)
0927 15:21:59.422316 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.422323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.432397 (+ 10074us) server_negotiation.cc:407] Connection header received
0927 15:21:59.432949 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.432958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.433052 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.433134 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:59.437880 (+ 4746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.439156 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.441346 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.442113 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.442235 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.453515 (+ 11280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:59.453541 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:59.453565 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:59.453637 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:59.458899 (+ 5262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.460243 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.460255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.460259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.460484 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.461037 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.461042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.461046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.462061 (+ 1015us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:59.462382 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.462742 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.463010 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:21:59.464321 14615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:59.580543 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 14487
I20260927 15:21:59.596923 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 14626
Sep 27 15:21:59 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:21:59.619575 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.598135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.2.246.66:34509)
0927 15:21:59.598359 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.598364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.598389 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:21:59.598602 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.598607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.598696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.598808 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:59.600436 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.601694 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.606230 (+ 4536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.606715 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.606878 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.612232 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:59.612250 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:59.612268 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:59.612315 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:59.615670 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.616542 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.616550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.616553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.616739 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.617478 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.617482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.617485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.617993 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:59.618239 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.618946 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.619142 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:21:59.620561 14758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:59.723301 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 14626
I20260927 15:21:59.745035 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 14769
Sep 27 15:21:59 dist-test-slave-14jq krb5kdc[13931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:21:59.764479 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:59.746139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.2.246.67:41903)
0927 15:21:59.746353 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:59.746358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:59.746476 (+ 118us) server_negotiation.cc:407] Connection header received
0927 15:21:59.746670 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:59.746675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:59.746834 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:59.746952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:59.749103 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.750182 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.751761 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:59.752187 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:59.752390 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:59.757762 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:59.757778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:59.757797 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:59.757910 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:59.761034 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.762177 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.762184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.762186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.762325 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:59.762850 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:59.762853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:59.762855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:59.763252 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:59.763472 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:59.763948 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:21:59.764161 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:21:59.765211 14892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:21:59.871192 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 14769
I20260927 15:21:59.888428 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 14375
I20260927 15:21:59.890108 14437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:22:00.015785 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 14375
I20260927 15:22:00.040889 3033 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:22:00.379128 3033 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:22:00.379446 3033 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:22:00.727789 3033 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:22:00.728 UTC [13961] LOG: received smart shutdown request
2026-09-27 15:22:00.732 UTC [13961] LOG: background worker "logical replication launcher" (PID 13970) exited with exit code 1
2026-09-27 15:22:00.732 UTC [13965] LOG: shutting down
2026-09-27 15:22:00.939 UTC [13961] LOG: database system is shut down
2026-09-27T15:22:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57404 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:22:01 dist-test-slave-14jq krb5kdc[14956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:22:01 dist-test-slave-14jq krb5kdc[14956](info): set up 2 sockets
Sep 27 15:22:01 dist-test-slave-14jq krb5kdc[14956](info): commencing operation
krb5kdc: starting...
W20260927 15:22:02.242892 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:22:02 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:22:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangeradmin_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangeradmin_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangerlookup_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangerlookup_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
I20260927 15:22:02.665134 3033 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbUGd5v/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/postgres -l logfile start
2026-09-27 15:22:05.715 UTC [14986] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:22:05.715 UTC [14986] LOG: listening on IPv4 address "127.2.246.84", port 41495
2026-09-27 15:22:05.719 UTC [14986] LOG: listening on Unix socket "/tmp/.s.PGSQL.41495"
2026-09-27 15:22:05.746 UTC [14989] LOG: database system was shut down at 2026-09-27 15:22:03 UTC
2026-09-27 15:22:05.750 UTC [14986] LOG: database system is ready to accept connections
I20260927 15:22:06.734112 3033 mini_postgres.cc:96] Postgres bound to 41495
2026-09-27 15:22:06.755 UTC [14997] FATAL: database "slave" does not exist
127.2.246.84:41495 - accepting connections
I20260927 15:22:06.756458 3033 mini_ranger.cc:161] Starting Ranger...
I20260927 15:22:06.787746 3033 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:22:07.359584 3033 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:22:07.359737 3033 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin
2026-09-27 15:22:07,705 [I] DB FLAVOR :POSTGRES
2026-09-27 15:22:07,706 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:22:07,706 [I] Checking connection..
2026-09-27 15:22:07,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:22:07,992 [I] Checking connection passed.
2026-09-27 15:22:07,993 [I] --------- Verifying version history table ---------
2026-09-27 15:22:07,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:22:08,297 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:22:08,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:22:08,590 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:22:08,590 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:22:08,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:22:08.881 UTC [15061] WARNING: there is no transaction in progress
2026-09-27 15:22:08,889 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:22:08,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:22:09,181 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:22:09,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:22:09,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:22:09,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:22:09,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-14jq.c.gcp-upstream.internal','N') ;"
2026-09-27 15:22:10,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:22:10,362 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:22:10,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:22:10,681 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:22:10,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:22:10,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:22:11,304 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:22:11,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:22:12.558 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.578 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.601 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.622 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.646 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.673 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.737 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.746 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.763 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.776 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.792 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.809 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.822 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.830 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.839 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:12.847 UTC [15169] WARNING: there is no transaction in progress
2026-09-27 15:22:13,725 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:22:13,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:22:13,999 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:22:13,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:22:14,268 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:22:14,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:22:14,604 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:22:14,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:22:14,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:22:15,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-14jq.c.gcp-upstream.internal';"
2026-09-27 15:22:15,445 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:22:15,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:22:15,730 [I] DB_PATCHES have already been applied
I20260927 15:22:15.738349 3033 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbUGd5v/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:22:15.738454 3033 mini_ranger.cc:193] Using host: 127.2.246.84
I20260927 15:22:15.757181 3033 mini_ranger.cc:227] Ranger admin URL: http://127.2.246.84:38985
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:22:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:22:16.972559 3033 mini_ranger.cc:160] Time spent starting Ranger: real 10.216s user 0.000s sys 0.047s
Sep 27, 2026 3:22:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 3:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:22:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:22:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:22:43.495966 3033 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangerkms_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/rangerkms_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/HTTP_127.2.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:22:43 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:22:43.692294 3033 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:22:43.718954 3033 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:22:44.734181 3033 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:22:44.734351 3033 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms
I20260927 15:22:44.734400 3033 mini_ranger_kms.cc:227] Using postgres at 127.2.246.84:41495
2026-09-27 15:22:45,149 [I] DB FLAVOR :POSTGRES
2026-09-27 15:22:45,149 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:22:45,150 [I] Checking connection
2026-09-27 15:22:45,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:22:45,458 [I] connection success
2026-09-27 15:22:45,459 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:22:45,459 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:22:45,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:22:45,768 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:22:45,769 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:22:45,769 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:22:45,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.246.84:41495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:22:46,077 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:22:46.659564 3033 mini_ranger_kms.cc:310] Created kms service
I20260927 15:22:46.862313 3033 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:22:46.909411 3033 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:22:46 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:47.440387 3033 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:22:47.440842 3033 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbUGd5v/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbUGd5v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:22:47.440907 3033 mini_ranger_kms.cc:240] Using host: 127.2.246.84
I20260927 15:22:47.461083 3033 mini_ranger_kms.cc:276] Ranger KMS PID: 15350
I20260927 15:22:47.461261 3033 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.246.84:44303
Sep 27, 2026 3:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:22:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:22:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:22:49.822854 3033 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.131s user 0.061s sys 0.375s
I20260927 15:22:49.823175 3033 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:22:49.823375 3033 mini_ranger_kms.cc:401] 127.2.246.84:44303/kms/v1/keys
Sep 27 15:22:49 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:22:50 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
Sep 27 15:22:50 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.246.84@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
Sep 27 15:22:53 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:22:53.406497 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44919
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44919
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:22:53.842337 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:53.842777 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:53.842983 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:53.852519 15400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:22:53.852695 15400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:53.852807 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:53.852921 15400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:22:53.853027 15400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:22:53.866168 15400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.246.84:44303/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44919
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:53.868013 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:53.869757 15400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:53.886282 15405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:53.886569 15406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:53.887347 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:53.887876 15400 server_base.cc:1034] running on GCE node
I20260927 15:22:53.889067 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:53.891209 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:53.892554 15400 hybrid_clock.cc:648] HybridClock initialized: now 1790522573892459 us; error 61 us; skew 500 ppm
I20260927 15:22:53.893052 15400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:53 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:53.897136 15400 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:22:53.899271 15400 webserver.cc:460] Webserver started at http://127.2.246.126:43295/ using document root <none> and password file <none>
I20260927 15:22:53.900182 15400 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:53.900331 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:22:53.900904 15400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:22:53 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:53.974006 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "ea2bfba34c7e462a9693d391b2ad71dd"
format_stamp: "Formatted at 2026-09-27 15:22:53 on dist-test-slave-14jq"
server_key: "71942a5dde4a54471a16aa379f5886b3"
server_key_iv: "f96faa6b143516d5ae942d35dc9fe7d5"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:53.975167 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "ea2bfba34c7e462a9693d391b2ad71dd"
format_stamp: "Formatted at 2026-09-27 15:22:53 on dist-test-slave-14jq"
server_key: "71942a5dde4a54471a16aa379f5886b3"
server_key_iv: "f96faa6b143516d5ae942d35dc9fe7d5"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:53.982306 15400 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 15:22:54.008388 15415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:54.011023 15400 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:22:54.011375 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "ea2bfba34c7e462a9693d391b2ad71dd"
format_stamp: "Formatted at 2026-09-27 15:22:53 on dist-test-slave-14jq"
server_key: "71942a5dde4a54471a16aa379f5886b3"
server_key_iv: "f96faa6b143516d5ae942d35dc9fe7d5"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:54.011682 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:54.055712 15400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:54.061573 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:54.062050 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:54.088868 15400 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44919
I20260927 15:22:54.088882 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44919 every 8 connection(s)
I20260927 15:22:54.091082 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:22:54.093621 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 15400
I20260927 15:22:54.093890 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:22:54.094751 3033 external_mini_cluster.cc:1428] Setting key 5bbe0077f4607e6d303c801db572ac99
I20260927 15:22:54.096544 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:22:54.111255 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Bootstrap starting.
Sep 27 15:22:54 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:22:54.115070 15467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Neither blocks nor log segments found. Creating new log.
I20260927 15:22:54.116600 15467 log.cc:824] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:22:54.121119 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: No bootstrap required, opened a new log
I20260927 15:22:54.123049 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:54.097277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.2.246.126:44919)
0927 15:22:54.098376 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:54.098392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:54.098443 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:22:54.100413 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:54.100443 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:54.100985 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:54.101474 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:54.103150 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:54.105260 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:54.106911 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:54.107986 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:54.108234 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:54.113399 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:54.113414 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:54.113434 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:54.113484 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:54.118102 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:54.119019 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:54.119026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:54.119034 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:54.119227 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:54.119646 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:54.119656 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:54.119658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:54.120298 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:54.120656 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:54.121147 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 15:22:54.121675 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":505,"threads_started":1}
I20260927 15:22:54.132112 15467 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:54.132814 15467 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:22:54.132961 15467 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2bfba34c7e462a9693d391b2ad71dd, State: Initialized, Role: FOLLOWER
I20260927 15:22:54.133746 15467 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:54.134111 15467 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:22:54.134287 15467 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:22:54.134497 15467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:22:54.137012 15467 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:54.137768 15467 leader_election.cc:302] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea2bfba34c7e462a9693d391b2ad71dd; no voters:
I20260927 15:22:54.138597 15467 leader_election.cc:288] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:22:54.138896 15472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:22:54.139761 15472 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 LEADER]: Becoming Leader. State: Replica: ea2bfba34c7e462a9693d391b2ad71dd, State: Running, Role: LEADER
I20260927 15:22:54.140498 15472 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:54.141184 15467 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:22:54.144405 15473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } } }
I20260927 15:22:54.144562 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea2bfba34c7e462a9693d391b2ad71dd. Latest consensus state: current_term: 1 leader_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } } }
I20260927 15:22:54.145637 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:22:54.144884 15473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:22:54.148365 15481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:22:54.156483 15481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:22:54.157599 15467 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:22:54.191924 15481 catalog_manager.cc:1347] Generated new cluster ID: 23708d8f716e43b4a021e58530231d92
I20260927 15:22:54.192220 15481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:22:54.220500 15481 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:22:54.222126 15481 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:22:54.230655 15481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Generated new TSK 0
I20260927 15:22:54.231631 15481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:22:55.103801 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:22:55 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:22:55.640056 15512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:55.640506 15512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:55.640764 15512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:55.652966 15512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:55.653133 15512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:55.653338 15512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:22:55.669983 15512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:55.671813 15512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:55.673576 15512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:55.694437 15523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:55.704031 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:55.704619 15512 server_base.cc:1034] running on GCE node
W20260927 15:22:55.705680 15522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:55.706967 15512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:55.710903 15512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:55.712443 15512 hybrid_clock.cc:648] HybridClock initialized: now 1790522575712273 us; error 160 us; skew 500 ppm
I20260927 15:22:55.712940 15512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:55 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:55.717170 15512 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:22:55.725528 15512 webserver.cc:460] Webserver started at http://127.2.246.65:44665/ using document root <none> and password file <none>
I20260927 15:22:55.726783 15512 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:55.727058 15512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:22:55.727659 15512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:22:55 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:55.753248 15512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b"
format_stamp: "Formatted at 2026-09-27 15:22:55 on dist-test-slave-14jq"
server_key: "18460db05472a51a26e7689d0ee1db24"
server_key_iv: "6769cf52aef5e2af53a82aeabe235686"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:55.754688 15512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b"
format_stamp: "Formatted at 2026-09-27 15:22:55 on dist-test-slave-14jq"
server_key: "18460db05472a51a26e7689d0ee1db24"
server_key_iv: "6769cf52aef5e2af53a82aeabe235686"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:55.764654 15512 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s
I20260927 15:22:55.801283 15532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:55.803870 15512 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.012s sys 0.004s
I20260927 15:22:55.804175 15512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b"
format_stamp: "Formatted at 2026-09-27 15:22:55 on dist-test-slave-14jq"
server_key: "18460db05472a51a26e7689d0ee1db24"
server_key_iv: "6769cf52aef5e2af53a82aeabe235686"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:55.804591 15512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:55.877076 15512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:55.883716 15512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:55.884389 15512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:55.898442 15512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:22:55.907279 15512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:22:55.907492 15512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 15:22:55.907764 15512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:22:55.907909 15512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:22:56 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:56.354470 15512 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:46467
I20260927 15:22:56.354487 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:46467 every 8 connection(s)
I20260927 15:22:56.356498 15512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:22:56.364737 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 15512
I20260927 15:22:56.365007 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:22:56.366027 3033 external_mini_cluster.cc:1428] Setting key 326c279a7e588f300ccd42b724cbf10e
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Sep 27 15:22:56 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:22:56.448920 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:56.374274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:45935 (local address 127.2.246.126:44919)
0927 15:22:56.398512 (+ 24238us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:56.398520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:56.398564 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:22:56.400158 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:56.400168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:56.400303 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:56.400426 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:56.403944 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:56.405297 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:56.418133 (+ 12836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:56.418970 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:56.419143 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:56.440413 (+ 21270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:56.440422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:56.440426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:56.440476 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:56.444350 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:56.445607 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:56.445613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:56.445616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:56.445802 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:56.446747 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:56.446751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:56.446754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:56.447119 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:56.447310 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:56.448313 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 15:22:56.448544 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24001,"thread_start_us":23757,"threads_started":1}
I20260927 15:22:56.451721 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:56.374717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.65:45935)
0927 15:22:56.395078 (+ 20361us) negotiation.cc:107] Waiting for socket to connect
0927 15:22:56.395191 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0927 15:22:56.399554 (+ 4363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:22:56.401283 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:22:56.401300 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:22:56.402293 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:22:56.403551 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:56.403620 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:56.405643 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:56.405650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:56.408255 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:56.408292 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:56.419760 (+ 11468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:56.419769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:56.420255 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:56.426344 (+ 6089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:22:56.426388 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:22:56.439991 (+ 13603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:22:56.444666 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:56.444678 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:56.444700 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:56.445445 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:56.445980 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:56.445984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:56.445987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:56.446586 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:56.447399 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:22:56.447407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:22:56.447890 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 15:22:56.449468 (+ 1578us) client_negotiation.cc:240] Negotiation successful
0927 15:22:56.450151 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19766,"thread_start_us":19430,"threads_started":1}
I20260927 15:22:56.454113 15645 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:22:56.454664 15645 heartbeater.cc:464] Registering TS with master...
I20260927 15:22:56.455811 15645 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:22:56.459408 15432 ts_manager.cc:194] Registered new tserver with Master: d36a2f9fa487446dac2a8cd4d8e6b47b (127.2.246.65:46467)
I20260927 15:22:56.462011 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:45935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
W20260927 15:22:56.490097 3033 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.246.66: real 0.120s user 0.005s sys 0.014s
I20260927 15:22:56.491017 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:22:56 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
W20260927 15:22:56.976274 15651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:56.976621 15651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:56.976787 15651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:56.984459 15651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:56.984644 15651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:56.984851 15651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:22:56.995311 15651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:56.996810 15651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:56.998268 15651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:57.014978 15665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:57.015131 15666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:57.016608 15668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:57.016933 15651 server_base.cc:1034] running on GCE node
I20260927 15:22:57.017735 15651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:57.019167 15651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:57.020434 15651 hybrid_clock.cc:648] HybridClock initialized: now 1790522577020312 us; error 112 us; skew 500 ppm
I20260927 15:22:57.020818 15651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:57.024847 15651 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:22:57.027103 15651 webserver.cc:460] Webserver started at http://127.2.246.66:42125/ using document root <none> and password file <none>
I20260927 15:22:57.028201 15651 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:57.028366 15651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:22:57.028780 15651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:57.050956 15651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "afb9a9952d52492eb703e23f2801a7a4"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "33b16a1f3a74f50f7a9f652d247a0351"
server_key_iv: "aa444c3106e6317ca3ffed223551a49a"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.052665 15651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "afb9a9952d52492eb703e23f2801a7a4"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "33b16a1f3a74f50f7a9f652d247a0351"
server_key_iv: "aa444c3106e6317ca3ffed223551a49a"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.061784 15651 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260927 15:22:57.083909 15675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.086297 15651 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:22:57.086611 15651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "afb9a9952d52492eb703e23f2801a7a4"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "33b16a1f3a74f50f7a9f652d247a0351"
server_key_iv: "aa444c3106e6317ca3ffed223551a49a"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.086962 15651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:57.117606 15651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:57.124512 15651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:57.124912 15651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:57.126202 15651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:22:57.127981 15651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:22:57.128070 15651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.128257 15651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:22:57.128326 15651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.171697 15651 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:41803
I20260927 15:22:57.171725 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:41803 every 8 connection(s)
I20260927 15:22:57.174139 15651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:22:57.184360 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 15651
I20260927 15:22:57.184569 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:22:57.185153 3033 external_mini_cluster.cc:1428] Setting key 199b4035105edf2550b54f070e50297b
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:22:57.201236 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:57.177785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:41945 (local address 127.2.246.126:44919)
0927 15:22:57.178463 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:57.178470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:57.179608 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 15:22:57.181579 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:57.181586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:57.181700 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:57.181930 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:57.183693 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.184809 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.187130 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.187732 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.187881 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:57.192454 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:57.192470 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:57.192475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:57.192545 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:57.196380 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.197634 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.197645 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.197652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.197922 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.198591 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.198599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.198605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.199057 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:57.199243 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:57.200358 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0927 15:22:57.200692 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
I20260927 15:22:57.202500 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:57.178450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.66:41945)
0927 15:22:57.179385 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 15:22:57.179434 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:22:57.181223 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:22:57.182276 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:22:57.182286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:22:57.182762 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:22:57.183451 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:57.183471 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.185060 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.185067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:57.186943 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:57.186958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.187976 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.187984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:57.188311 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:57.189230 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:22:57.189265 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:22:57.192098 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:22:57.196648 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:57.196661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:57.196686 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:57.197397 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:57.198068 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:57.198073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:57.198076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:57.198378 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:57.199232 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:22:57.199242 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:22:57.199837 (+ 595us) client_negotiation.cc:769] Sending connection context
0927 15:22:57.200180 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 15:22:57.200840 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":403,"threads_started":1}
I20260927 15:22:57.204829 15788 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:22:57.205251 15788 heartbeater.cc:464] Registering TS with master...
I20260927 15:22:57.206332 15788 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:22:57.208655 15432 ts_manager.cc:194] Registered new tserver with Master: afb9a9952d52492eb703e23f2801a7a4 (127.2.246.66:41803)
I20260927 15:22:57.210122 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:41945
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:22:57.245201 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:22:57.467181 15645 heartbeater.cc:503] Master 127.2.246.126:44919 was elected leader, sending a full tablet report...
W20260927 15:22:57.673369 15794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:57.673771 15794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:57.674015 15794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:57.682816 15794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:57.683022 15794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:57.683287 15794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:22:57.695547 15794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:57.697321 15794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:57.699054 15794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:57.715700 15800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:57.715873 15799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:57.715896 15802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:57.717316 15794 server_base.cc:1034] running on GCE node
I20260927 15:22:57.718238 15794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:57.719542 15794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:57.720871 15794 hybrid_clock.cc:648] HybridClock initialized: now 1790522577720783 us; error 53 us; skew 500 ppm
I20260927 15:22:57.721354 15794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:57.732383 15794 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:22:57.734771 15794 webserver.cc:460] Webserver started at http://127.2.246.67:34335/ using document root <none> and password file <none>
I20260927 15:22:57.735992 15794 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:57.736251 15794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:22:57.736759 15794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:57.758045 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "e99b233eb3224036ba6fa2ddd5a57983"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "895d1785b7692e600e5128aabb874eb1"
server_key_iv: "0299170ae7e20f0d335dd22a44da86e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.759305 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "e99b233eb3224036ba6fa2ddd5a57983"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "895d1785b7692e600e5128aabb874eb1"
server_key_iv: "0299170ae7e20f0d335dd22a44da86e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.767652 15794 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260927 15:22:57.788587 15809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.790395 15794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:22:57.790598 15794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "e99b233eb3224036ba6fa2ddd5a57983"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "895d1785b7692e600e5128aabb874eb1"
server_key_iv: "0299170ae7e20f0d335dd22a44da86e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:57.790829 15794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:57.835757 15794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:57.840698 15794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:57.841092 15794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:57.842535 15794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:22:57.844484 15794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:22:57.844619 15794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.844775 15794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:22:57.844848 15794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:57.886953 15794 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:45177
I20260927 15:22:57.887061 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:45177 every 8 connection(s)
I20260927 15:22:57.888474 15794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:22:57.895167 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 15794
I20260927 15:22:57.895413 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:22:57.896034 3033 external_mini_cluster.cc:1428] Setting key a3773daf9d43044a247b028091ad649b
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:22:57.918193 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:57.892379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:56919 (local address 127.2.246.126:44919)
0927 15:22:57.893029 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:57.893035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:57.894240 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 15:22:57.897212 (+ 2972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:57.897218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:57.897334 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:57.897485 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:57.899832 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.900680 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.902719 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.903255 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.903446 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:57.909345 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:57.909356 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:57.909360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:57.909420 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:57.913674 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.914748 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.914753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.914756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.914981 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.915490 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.915509 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.915511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.916033 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:57.916196 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:57.917193 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 15:22:57.917582 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":365,"threads_started":1}
I20260927 15:22:57.919143 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:57.892926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.67:56919)
0927 15:22:57.894016 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 15:22:57.894083 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:22:57.896839 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:22:57.897830 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:22:57.897846 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:22:57.898578 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:22:57.899564 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:57.899591 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.900897 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.900902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:57.902520 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:57.902538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.903591 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.903603 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:57.903938 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:57.904893 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:22:57.904926 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:22:57.908810 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:22:57.913959 (+ 5149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:57.913970 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:57.913988 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:57.914552 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:57.915100 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:57.915104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:57.915106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:57.915322 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:57.916247 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:22:57.916265 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:22:57.916727 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 15:22:57.917001 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 15:22:57.917541 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":390,"threads_started":1}
I20260927 15:22:57.921321 15922 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:22:57.921723 15922 heartbeater.cc:464] Registering TS with master...
I20260927 15:22:57.922713 15922 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:22:57.924855 15432 ts_manager.cc:194] Registered new tserver with Master: e99b233eb3224036ba6fa2ddd5a57983 (127.2.246.67:45177)
I20260927 15:22:57.926192 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:56919
I20260927 15:22:57.927294 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:22:57.929236 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 15512
Sep 27 15:22:57 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:22:57.950399 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:57.930429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.2.246.65:46467)
0927 15:22:57.931173 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:57.931187 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:57.931216 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:22:57.931280 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:57.931286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:57.931489 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:57.931643 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:57.933571 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.934818 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.936745 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:57.937339 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:57.937480 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:57.944012 (+ 6532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:57.944027 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:57.944051 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:57.944096 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:57.947171 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.948292 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.948298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.948301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.948446 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:57.948814 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:57.948817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:57.948819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:57.949165 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:57.949473 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:57.949898 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 15:22:57.950110 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":431,"threads_started":1}
I20260927 15:22:57.952121 15640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:22:58.050119 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 15512
I20260927 15:22:58.067008 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 15651
Sep 27 15:22:58 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:22:58.095857 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:58.068253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.2.246.66:41803)
0927 15:22:58.068860 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:58.068868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:58.068902 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:22:58.068972 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:58.068979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:58.069209 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:58.069391 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:58.071346 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:58.072365 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:58.074746 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:58.075428 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:58.075572 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:58.087393 (+ 11821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:58.087425 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:58.087444 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:58.087513 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:58.091277 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:58.092720 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:58.092730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:58.092732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:58.092930 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:58.093606 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:58.093610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:58.093613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:58.094129 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:58.094442 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:58.094994 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 15:22:58.095381 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":333,"threads_started":1}
I20260927 15:22:58.097476 15783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:22:58.184872 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 15651
I20260927 15:22:58.203663 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 15794
Sep 27 15:22:58 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:22:58.224593 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:58.205184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.2.246.67:45177)
0927 15:22:58.205977 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:58.205985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:58.206021 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:22:58.206093 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:58.206100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:58.206330 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:58.206488 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:58.208138 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:58.209341 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:58.211045 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:58.211692 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:58.211841 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:58.217275 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:58.217291 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:58.217310 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:58.217380 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:58.220843 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:58.222201 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:58.222209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:58.222211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:58.222371 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:58.222931 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:58.222935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:58.222938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:58.223311 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:58.223617 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:58.224061 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:22:58.224286 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":356,"threads_started":1}
I20260927 15:22:58.226591 15917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:22:58.324378 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 15794
I20260927 15:22:58.344213 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 15400
I20260927 15:22:58.346133 15462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:22:58.458180 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 15400
I20260927 15:22:58.483704 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44919
--webserver_interface=127.2.246.126
--webserver_port=43295
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44919
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:22:58.929128 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:58.929551 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:58.929747 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:58.938555 15934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:22:58.938719 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:58.938803 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:58.938902 15934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:22:58.938989 15934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:22:58.954025 15934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44919
--ranger_config_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=43295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:58.955900 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:58.957808 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:58.974247 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:58.974236 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:58.974148 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:58.975701 15934 server_base.cc:1034] running on GCE node
I20260927 15:22:58.976902 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:58.979164 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:58.980530 15934 hybrid_clock.cc:648] HybridClock initialized: now 1790522578980457 us; error 51 us; skew 500 ppm
I20260927 15:22:58.981076 15934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:58 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522578, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:58.985755 15934 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:22:58.988025 15934 webserver.cc:460] Webserver started at http://127.2.246.126:43295/ using document root <none> and password file <none>
I20260927 15:22:58.988953 15934 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:58.989140 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:22:59 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522578, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:59.046824 15934 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:22:59.050895 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:59.053010 15934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:22:59.053222 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "ea2bfba34c7e462a9693d391b2ad71dd"
format_stamp: "Formatted at 2026-09-27 15:22:53 on dist-test-slave-14jq"
server_key: "71942a5dde4a54471a16aa379f5886b3"
server_key_iv: "f96faa6b143516d5ae942d35dc9fe7d5"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:59.054358 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:59.072690 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:59.077297 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:59.077556 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:59.099339 15934 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44919
I20260927 15:22:59.099438 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44919 every 8 connection(s)
I20260927 15:22:59.101006 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:22:59.103929 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 15934
I20260927 15:22:59.104815 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:46467
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:44919
--webserver_port=44665
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:22:59.111630 16002 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:22:59.116006 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Bootstrap starting.
I20260927 15:22:59.141635 16002 log.cc:824] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:22:59.153083 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:22:59.153765 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Bootstrap complete.
I20260927 15:22:59.166790 16002 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:59.168126 16002 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2bfba34c7e462a9693d391b2ad71dd, State: Initialized, Role: FOLLOWER
I20260927 15:22:59.168995 16002 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:59.169286 16002 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:22:59.169428 16002 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:22:59.169629 16002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:22:59.174971 16002 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:59.175740 16002 leader_election.cc:302] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea2bfba34c7e462a9693d391b2ad71dd; no voters:
I20260927 15:22:59.176625 16002 leader_election.cc:288] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:22:59.176807 16006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:22:59.178459 16006 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [term 2 LEADER]: Becoming Leader. State: Replica: ea2bfba34c7e462a9693d391b2ad71dd, State: Running, Role: LEADER
I20260927 15:22:59.179209 16006 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } }
I20260927 15:22:59.179734 16002 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:22:59.182542 16008 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea2bfba34c7e462a9693d391b2ad71dd. Latest consensus state: current_term: 2 leader_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } } }
I20260927 15:22:59.183158 16008 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:22:59.183152 16007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2bfba34c7e462a9693d391b2ad71dd" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44919 } } }
I20260927 15:22:59.183713 16007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:22:59.185473 16012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:22:59.205188 16012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:22:59.207158 16012 catalog_manager.cc:1259] Loaded cluster ID: 23708d8f716e43b4a021e58530231d92
I20260927 15:22:59.207391 16012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:22:59.212204 16012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:22:59.214219 16012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Loaded TSK: 0
I20260927 15:22:59.216279 16012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:22:59.224059 16002 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 15:22:59.600085 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:22:59.600728 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:22:59.601015 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:22:59.613799 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:22:59.614082 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:22:59.614339 16004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:22:59.631130 16004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=44665
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:22:59.633323 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:22:59.635277 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:22:59.668313 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:59.670738 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:22:59.674427 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:22:59.676657 16004 server_base.cc:1034] running on GCE node
I20260927 15:22:59.677872 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:22:59.681233 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:22:59.684767 16004 hybrid_clock.cc:648] HybridClock initialized: now 1790522579684641 us; error 90 us; skew 500 ppm
I20260927 15:22:59.685436 16004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:22:59 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:22:59.690474 16004 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:22:59.693204 16004 webserver.cc:460] Webserver started at http://127.2.246.65:44665/ using document root <none> and password file <none>
I20260927 15:22:59.694435 16004 fs_manager.cc:359] Metadata directory not provided
I20260927 15:22:59.694643 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:22:59 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:22:59.728943 16004 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s
I20260927 15:22:59.741432 16050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:59.743865 16004 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.006s
I20260927 15:22:59.744115 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b"
format_stamp: "Formatted at 2026-09-27 15:22:55 on dist-test-slave-14jq"
server_key: "18460db05472a51a26e7689d0ee1db24"
server_key_iv: "6769cf52aef5e2af53a82aeabe235686"
server_key_version: "kuduclusterkey@0"
I20260927 15:22:59.745682 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:22:59.809499 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:22:59.815665 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:22:59.816007 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:22:59.823472 16004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:22:59.831957 16004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:22:59.832119 16004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:59.832243 16004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:22:59.832326 16004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:22:59.938742 16004 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:46467
I20260927 15:22:59.938786 16163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:46467 every 8 connection(s)
I20260927 15:22:59.940693 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:22:59.944926 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16004
I20260927 15:22:59.946000 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:41803
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:44919
--webserver_port=42125
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:22:59 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:22:59.998458 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:59.948118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.65:50989)
0927 15:22:59.957264 (+ 9146us) negotiation.cc:107] Waiting for socket to connect
0927 15:22:59.957354 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 15:22:59.960028 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:22:59.963113 (+ 3085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:22:59.963128 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:22:59.963895 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:22:59.964878 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:59.964914 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:59.967828 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:59.967835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:59.970086 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:22:59.970106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:22:59.972583 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:22:59.972595 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:22:59.972935 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:59.977017 (+ 4082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:22:59.977055 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:22:59.980610 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:22:59.991328 (+ 10718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:59.991343 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:59.991367 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:59.992212 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:59.993168 (+ 956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:22:59.993176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:22:59.993180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:22:59.993724 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:22:59.995197 (+ 1473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:22:59.995210 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:22:59.995824 (+ 614us) client_negotiation.cc:769] Sending connection context
0927 15:22:59.996256 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 15:22:59.996922 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8698,"thread_start_us":8415,"threads_started":1}
I20260927 15:23:00.001561 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:22:59.946998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:50989 (local address 127.2.246.126:44919)
0927 15:22:59.958669 (+ 11671us) server_negotiation.cc:206] Beginning negotiation
0927 15:22:59.958696 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 15:22:59.958819 (+ 123us) server_negotiation.cc:407] Connection header received
0927 15:22:59.960997 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:22:59.961026 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:22:59.961729 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:22:59.962315 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:22:59.965295 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:59.967539 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:59.970369 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:22:59.972174 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:22:59.972536 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:22:59.985557 (+ 13021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:22:59.985573 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:22:59.985606 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:22:59.985688 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:22:59.990928 (+ 5240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:59.992556 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:59.992570 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:59.992584 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:59.992920 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:22:59.993932 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:22:59.993939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:22:59.993942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:22:59.994824 (+ 882us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:22:59.995069 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:22:59.999634 (+ 4565us) server_negotiation.cc:299] Negotiation successful
0927 15:23:00.000133 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11233,"thread_start_us":10923,"threads_started":1}
I20260927 15:23:00.007436 16164 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:23:00.008159 16164 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:00.009398 16164 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
W20260927 15:23:00.014945 16164 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:44919 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:23:00.015281 16164 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:00.015672 16164 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:23:00.017656 16164 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:00.018236 16164 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
W20260927 15:23:00.020495 16164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:23:00.402789 16168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:00.403249 16168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:00.403493 16168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:00.415518 16168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:00.415750 16168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:00.415961 16168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:23:00.427275 16168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:41803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=42125
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:00.428781 16168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:00.430212 16168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:00.458982 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:00.459026 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:00.459085 16185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:00.461449 16168 server_base.cc:1034] running on GCE node
I20260927 15:23:00.462363 16168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:00.463742 16168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:00.467154 16168 hybrid_clock.cc:648] HybridClock initialized: now 1790522580467051 us; error 105 us; skew 500 ppm
I20260927 15:23:00.467595 16168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:00 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:00.471974 16168 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:23:00.473924 16168 webserver.cc:460] Webserver started at http://127.2.246.66:42125/ using document root <none> and password file <none>
I20260927 15:23:00.474714 16168 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:00.474879 16168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:23:00 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:23:00.499207 16168 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:23:00.502437 16193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:00.505302 16168 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 15:23:00.505627 16168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "afb9a9952d52492eb703e23f2801a7a4"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "33b16a1f3a74f50f7a9f652d247a0351"
server_key_iv: "aa444c3106e6317ca3ffed223551a49a"
server_key_version: "kuduclusterkey@0"
I20260927 15:23:00.507615 16168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:00.560529 16168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:00.569476 16168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:00.569900 16168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:00.590462 16168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:00.601748 16168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:00.601941 16168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 15:23:00.602110 16168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:00.602202 16168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:00.659915 16168 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:41803
I20260927 15:23:00.660027 16307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:41803 every 8 connection(s)
I20260927 15:23:00.661886 16168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:23:00.664798 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16168
I20260927 15:23:00.665817 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.246.84:44303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:45177
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:44919
--webserver_port=34335
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:23:00 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:23:00 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:00.709935 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:00.668423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:60557 (local address 127.2.246.126:44919)
0927 15:23:00.669241 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:00.669252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:00.683300 (+ 14048us) server_negotiation.cc:407] Connection header received
0927 15:23:00.688387 (+ 5087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:00.688394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:00.688521 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:00.688616 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:00.691806 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:00.693031 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:00.695772 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:00.696447 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:00.696563 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:00.701392 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:00.701402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:00.701408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:00.701454 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:00.705205 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:00.706320 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:00.706331 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:00.706336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:00.706514 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:00.707163 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:00.707169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:00.707172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:00.707606 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:00.707779 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:00.708973 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0927 15:23:00.709270 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":669,"threads_started":1}
I20260927 15:23:00.712999 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:00.666756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.66:60557)
0927 15:23:00.682981 (+ 16225us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:00.683077 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:00.687064 (+ 3987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:00.689287 (+ 2223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:00.689300 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:00.690597 (+ 1297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:00.691536 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:00.691565 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:00.693262 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:00.693267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:00.695570 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:00.695588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:00.696981 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:00.696988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:00.697288 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:00.698184 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:00.698218 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:00.701071 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:00.705438 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:00.705449 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:00.705470 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:00.706159 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:00.706705 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:00.706711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:00.706715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:00.707017 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:00.707895 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:00.707906 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:00.708462 (+ 556us) client_negotiation.cc:769] Sending connection context
0927 15:23:00.710588 (+ 2126us) client_negotiation.cc:240] Negotiation successful
0927 15:23:00.711319 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15669,"thread_start_us":15310,"threads_started":1}
I20260927 15:23:00.715914 16308 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:23:00.716254 16308 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:00.717114 16308 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:23:00.719322 15967 ts_manager.cc:194] Registered new tserver with Master: afb9a9952d52492eb703e23f2801a7a4 (127.2.246.66:41803)
I20260927 15:23:00.721529 15967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:60557
I20260927 15:23:01.020635 16164 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:01.021121 16164 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:23:01.022830 15967 ts_manager.cc:194] Registered new tserver with Master: d36a2f9fa487446dac2a8cd4d8e6b47b (127.2.246.65:46467)
I20260927 15:23:01.024096 15967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:50989
W20260927 15:23:01.098063 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:01.098528 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:01.098711 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:01.110489 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:01.110736 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:01.110939 16312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:23:01.126246 16312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.246.84:44303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:45177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=34335
--tserver_master_addrs=127.2.246.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:01.128235 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:01.130542 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:01.153108 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:01.154471 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:01.164057 16312 server_base.cc:1034] running on GCE node
W20260927 15:23:01.165004 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:01.166093 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:01.171641 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:01.175177 16312 hybrid_clock.cc:648] HybridClock initialized: now 1790522581175059 us; error 114 us; skew 500 ppm
I20260927 15:23:01.175721 16312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:01 dist-test-slave-14jq krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:01.181099 16312 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:23:01.184188 16312 webserver.cc:460] Webserver started at http://127.2.246.67:34335/ using document root <none> and password file <none>
I20260927 15:23:01.185428 16312 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:01.185611 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:23:01 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:23:01.223404 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.007s sys 0.001s
I20260927 15:23:01.228163 16329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:01.230674 16312 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:23:01.230967 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "e99b233eb3224036ba6fa2ddd5a57983"
format_stamp: "Formatted at 2026-09-27 15:22:57 on dist-test-slave-14jq"
server_key: "895d1785b7692e600e5128aabb874eb1"
server_key_iv: "0299170ae7e20f0d335dd22a44da86e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:23:01.233559 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 15:23:01 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:23:01.272675 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:01.280306 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:01.280754 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:01.283926 16312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:01.299347 16312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:01.299575 16312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:01.299731 16312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:01.299813 16312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:01.455862 16312 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:45177
I20260927 15:23:01.455935 16441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:45177 every 8 connection(s)
I20260927 15:23:01.458591 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:23:01.462366 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16312
Sep 27 15:23:01 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:01.515573 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.463759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.2.246.126:44919)
0927 15:23:01.482314 (+ 18555us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.482324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.488331 (+ 6007us) server_negotiation.cc:407] Connection header received
0927 15:23:01.488612 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.488622 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.488772 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.488917 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:01.496935 (+ 8018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.498276 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.500394 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.501153 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.501289 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.505437 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:01.505448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:01.505454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:01.505515 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:01.510056 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.511181 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.511189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.511194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.511392 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.512575 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.512583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.512586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.513090 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:01.513445 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.514729 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.515092 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18354,"thread_start_us":18552,"threads_started":1}
I20260927 15:23:01.525460 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.470129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44919 (local address 127.2.246.67:42813)
0927 15:23:01.482552 (+ 12423us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.482635 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.485572 (+ 2937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.491949 (+ 6377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.491970 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.493026 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:01.494688 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.494734 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.496936 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.496943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.499598 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.499627 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.501232 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.501244 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.501634 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.502922 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:01.502960 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:01.510877 (+ 7917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:01.518558 (+ 7681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:01.518575 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:01.518605 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:01.519752 (+ 1147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:01.520352 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:01.520359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:01.520364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:01.520774 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:01.521648 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:01.521661 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:01.522359 (+ 698us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.522758 (+ 399us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.523550 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11939,"thread_start_us":16388,"threads_started":1}
I20260927 15:23:01.527443 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.482598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:42813 (local address 127.2.246.126:44919)
0927 15:23:01.490807 (+ 8209us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.490814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.490836 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:23:01.490910 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.490915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.491017 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.495032 (+ 4015us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:01.495345 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.496600 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.499891 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.500596 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.500741 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.511702 (+ 10961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:01.511710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:01.511716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:01.511760 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:01.518243 (+ 6483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.519966 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.519976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.519980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.520181 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.520943 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.520950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.520953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.521393 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:01.521558 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.526583 (+ 5025us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.526962 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8013,"thread_start_us":8594,"threads_started":1}
I20260927 15:23:01.529326 16442 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44919
I20260927 15:23:01.529878 16442 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:01.531212 16442 heartbeater.cc:511] Master 127.2.246.126:44919 requested a full tablet report, sending...
I20260927 15:23:01.534319 15967 ts_manager.cc:194] Registered new tserver with Master: e99b233eb3224036ba6fa2ddd5a57983 (127.2.246.67:45177)
I20260927 15:23:01.535665 15967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:42813
I20260927 15:23:01.538017 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:23:01.618165 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.596599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.2.246.126:44919)
0927 15:23:01.596872 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.596881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.597525 (+ 644us) server_negotiation.cc:407] Connection header received
0927 15:23:01.597929 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.597939 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.598060 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.598152 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:01.600619 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.601857 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.604110 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.604823 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.604949 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.609242 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:01.609250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:01.609256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:01.609318 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:01.613635 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.614903 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.614910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.614914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.615156 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:01.615687 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:01.615692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:01.615695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:01.616125 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:01.616430 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.617314 (+ 884us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.617687 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:23:01.628240 15967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:23:01.641973 15967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:23:01.727317 16308 heartbeater.cc:503] Master 127.2.246.126:44919 was elected leader, sending a full tablet report...
I20260927 15:23:01.728076 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.705104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:43058)
0927 15:23:01.718877 (+ 13773us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.718946 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.719224 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.720026 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.720034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.720073 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:01.720499 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.720513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.722362 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.722368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.724787 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.724806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.726832 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.726838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.727078 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.727134 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.727289 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.727665 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13377,"threads_started":1}
I20260927 15:23:01.728559 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.705204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.2.246.65:46467)
0927 15:23:01.714490 (+ 9286us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.714500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.719090 (+ 4590us) server_negotiation.cc:407] Connection header received
0927 15:23:01.719490 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.719498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.719795 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.720002 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:01.720670 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.722241 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.724952 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.726574 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.726745 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.727911 (+ 1166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.728001 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.728221 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8976,"thread_start_us":12616,"threads_started":1}
I20260927 15:23:01.762313 16099 tablet_service.cc:1467] Processing CreateTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 (DEFAULT_TABLE table=test-table [id=93cdab270fce4b21bc2df069437a7b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:01.764436 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc51d82ce9a4ac284d3c0a07bbc6326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:01.776185 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.710672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:45177 (local address 127.0.0.1:52118)
0927 15:23:01.728562 (+ 17890us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.728585 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.728876 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.754710 (+ 25834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.754717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.754772 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:01.755030 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.755044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.763512 (+ 8468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.763521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.765571 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.765599 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.775171 (+ 9572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.775180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.775466 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.775511 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.775597 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.775801 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17762,"mutex_wait_us":1,"thread_start_us":7644,"threads_started":1}
I20260927 15:23:01.777329 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.751694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41803 (local address 127.0.0.1:51630)
0927 15:23:01.752546 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.752571 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.752781 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.761573 (+ 8792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.761580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.761623 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:01.762089 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.762106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.767648 (+ 5542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.767654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.771424 (+ 3770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.771446 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.775224 (+ 3778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.775232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.776780 (+ 1548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.776816 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.776905 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.777038 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1310}
I20260927 15:23:01.778818 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.711236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.2.246.67:45177)
0927 15:23:01.740260 (+ 29024us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.740277 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.750420 (+ 10143us) server_negotiation.cc:407] Connection header received
0927 15:23:01.753025 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.753055 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.754321 (+ 1266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.754679 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:01.758082 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.759877 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.766054 (+ 6177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.767693 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.777909 (+ 10216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.778232 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.778320 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.778503 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28665,"thread_start_us":29121,"threads_started":1}
I20260927 15:23:01.779343 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.742688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.2.246.66:41803)
0927 15:23:01.760420 (+ 17732us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.760431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.760475 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:23:01.760626 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.760635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.760954 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.761242 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:01.762918 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.764549 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.771692 (+ 7143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.774898 (+ 3206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.775392 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.775685 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.777259 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.777515 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17498,"thread_start_us":19571,"threads_started":1}
I20260927 15:23:01.783468 16377 tablet_service.cc:1467] Processing CreateTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 (DEFAULT_TABLE table=test-table [id=93cdab270fce4b21bc2df069437a7b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:01.783849 16243 tablet_service.cc:1467] Processing CreateTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 (DEFAULT_TABLE table=test-table [id=93cdab270fce4b21bc2df069437a7b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:01.785167 16377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc51d82ce9a4ac284d3c0a07bbc6326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:01.785588 16243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc51d82ce9a4ac284d3c0a07bbc6326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:01.815871 16464 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Bootstrap starting.
Sep 27 15:23:01 dist-test-slave-14jq krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for HTTP/127.2.246.84@KRBTEST.COM
I20260927 15:23:01.820015 16464 tablet_bootstrap.cc:654] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:01.821415 16464 log.cc:824] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:01.831833 16466 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Bootstrap starting.
I20260927 15:23:01.845691 16465 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Bootstrap starting.
I20260927 15:23:01.851694 16466 tablet_bootstrap.cc:654] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:01.853583 16464 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: No bootstrap required, opened a new log
I20260927 15:23:01.853729 16466 log.cc:824] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:01.853989 16464 ts_tablet_manager.cc:1397] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.005s
I20260927 15:23:01.858878 16465 tablet_bootstrap.cc:654] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:01.860978 16465 log.cc:824] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:01.865418 16464 raft_consensus.cc:348] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.865950 16464 raft_consensus.cc:374] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:01.866075 16464 raft_consensus.cc:729] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99b233eb3224036ba6fa2ddd5a57983, State: Initialized, Role: FOLLOWER
I20260927 15:23:01.866268 16466 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: No bootstrap required, opened a new log
I20260927 15:23:01.866758 16466 ts_tablet_manager.cc:1397] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.000s
I20260927 15:23:01.870412 16464 consensus_queue.cc:260] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.874003 16465 tablet_bootstrap.cc:492] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: No bootstrap required, opened a new log
I20260927 15:23:01.874459 16465 ts_tablet_manager.cc:1397] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.005s
I20260927 15:23:01.895711 16464 ts_tablet_manager.cc:1428] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Time spent starting tablet: real 0.042s user 0.020s sys 0.016s
I20260927 15:23:01.896802 16466 raft_consensus.cc:348] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.897639 16466 raft_consensus.cc:374] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:01.897787 16466 raft_consensus.cc:729] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36a2f9fa487446dac2a8cd4d8e6b47b, State: Initialized, Role: FOLLOWER
I20260927 15:23:01.898751 16442 heartbeater.cc:503] Master 127.2.246.126:44919 was elected leader, sending a full tablet report...
I20260927 15:23:01.903029 16465 raft_consensus.cc:348] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.903784 16465 raft_consensus.cc:374] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:01.903930 16465 raft_consensus.cc:729] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb9a9952d52492eb703e23f2801a7a4, State: Initialized, Role: FOLLOWER
I20260927 15:23:01.904816 16465 consensus_queue.cc:260] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.906121 16466 consensus_queue.cc:260] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.915606 16164 heartbeater.cc:503] Master 127.2.246.126:44919 was elected leader, sending a full tablet report...
I20260927 15:23:01.915714 16465 ts_tablet_manager.cc:1428] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Time spent starting tablet: real 0.041s user 0.018s sys 0.001s
I20260927 15:23:01.916517 16470 raft_consensus.cc:482] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:23:01.916884 16470 raft_consensus.cc:504] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.923338 16470 leader_election.cc:288] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d36a2f9fa487446dac2a8cd4d8e6b47b (127.2.246.65:46467), afb9a9952d52492eb703e23f2801a7a4 (127.2.246.66:41803)
I20260927 15:23:01.927851 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.919835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.2.246.67:44613)
0927 15:23:01.920373 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.920400 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.920559 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.921714 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.921720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.921760 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:01.922011 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.922025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.923467 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.923473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.925272 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.925287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.927060 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.927067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.927253 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.927288 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.927387 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.927537 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380}
I20260927 15:23:01.928740 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.920929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:44613 (local address 127.2.246.65:46467)
0927 15:23:01.921197 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.921205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.921231 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:23:01.921304 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.921309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.921389 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.921498 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:01.922215 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.923269 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.925459 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.926691 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.926868 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.926947 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.928223 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.928414 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:23:01.936842 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.922238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:39615 (local address 127.2.246.66:41803)
0927 15:23:01.922492 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:01.922502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:01.929462 (+ 6960us) server_negotiation.cc:407] Connection header received
0927 15:23:01.929623 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:01.929629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:01.929750 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:01.929956 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:01.930470 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.931612 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.934225 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.935477 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.935633 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.935734 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:01.936395 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 15:23:01.936545 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:23:01.937747 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:01.921596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41803 (local address 127.2.246.67:39615)
0927 15:23:01.929289 (+ 7693us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:01.929312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:01.929456 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:01.930002 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:01.930008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:01.930039 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:01.930239 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.930249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.931952 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.931958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.934040 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:01.934056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:01.935983 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:01.935990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:01.936201 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:01.936236 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:23:01.937265 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0927 15:23:01.937417 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7566,"threads_started":1}
W20260927 15:23:01.939415 16309 tablet.cc:2369] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:01.941123 16466 ts_tablet_manager.cc:1428] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Time spent starting tablet: real 0.074s user 0.020s sys 0.000s
W20260927 15:23:01.945596 16165 tablet.cc:2369] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:01.951900 16263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc51d82ce9a4ac284d3c0a07bbc6326" candidate_uuid: "e99b233eb3224036ba6fa2ddd5a57983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb9a9952d52492eb703e23f2801a7a4" is_pre_election: true
I20260927 15:23:01.952525 16263 raft_consensus.cc:2454] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99b233eb3224036ba6fa2ddd5a57983 in term 0.
I20260927 15:23:01.952550 16119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc51d82ce9a4ac284d3c0a07bbc6326" candidate_uuid: "e99b233eb3224036ba6fa2ddd5a57983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" is_pre_election: true
I20260927 15:23:01.953294 16119 raft_consensus.cc:2454] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99b233eb3224036ba6fa2ddd5a57983 in term 0.
I20260927 15:23:01.954571 16333 leader_election.cc:302] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb9a9952d52492eb703e23f2801a7a4, e99b233eb3224036ba6fa2ddd5a57983; no voters:
I20260927 15:23:01.955214 16470 raft_consensus.cc:2781] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:23:01.955523 16470 raft_consensus.cc:482] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:23:01.955686 16470 raft_consensus.cc:3037] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:01.959177 16470 raft_consensus.cc:504] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:01.960144 16470 leader_election.cc:288] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [CANDIDATE]: Term 1 election: Requested vote from peers d36a2f9fa487446dac2a8cd4d8e6b47b (127.2.246.65:46467), afb9a9952d52492eb703e23f2801a7a4 (127.2.246.66:41803)
I20260927 15:23:01.960963 16119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc51d82ce9a4ac284d3c0a07bbc6326" candidate_uuid: "e99b233eb3224036ba6fa2ddd5a57983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b"
I20260927 15:23:01.961217 16119 raft_consensus.cc:3037] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:01.961508 16263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc51d82ce9a4ac284d3c0a07bbc6326" candidate_uuid: "e99b233eb3224036ba6fa2ddd5a57983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb9a9952d52492eb703e23f2801a7a4"
I20260927 15:23:01.961872 16263 raft_consensus.cc:3037] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:01.964900 16119 raft_consensus.cc:2454] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99b233eb3224036ba6fa2ddd5a57983 in term 1.
I20260927 15:23:01.966154 16263 raft_consensus.cc:2454] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99b233eb3224036ba6fa2ddd5a57983 in term 1.
I20260927 15:23:01.966938 16333 leader_election.cc:302] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb9a9952d52492eb703e23f2801a7a4, e99b233eb3224036ba6fa2ddd5a57983; no voters:
I20260927 15:23:01.967506 16470 raft_consensus.cc:2781] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 15:23:01.987599 16446 tablet.cc:2369] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:01.989106 16470 raft_consensus.cc:686] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 1 LEADER]: Becoming Leader. State: Replica: e99b233eb3224036ba6fa2ddd5a57983, State: Running, Role: LEADER
I20260927 15:23:01.989760 16470 consensus_queue.cc:237] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } }
I20260927 15:23:02.007714 15967 catalog_manager.cc:5495] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 reported cstate change: term changed from 0 to 1, leader changed from <none> to e99b233eb3224036ba6fa2ddd5a57983 (127.2.246.67). New cstate: current_term: 1 leader_uuid: "e99b233eb3224036ba6fa2ddd5a57983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e99b233eb3224036ba6fa2ddd5a57983" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 45177 } health_report { overall_health: HEALTHY } } }
I20260927 15:23:02.052246 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.042570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.2.246.67:45177)
0927 15:23:02.043221 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:02.043477 (+ 256us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:02.043528 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:23:02.043859 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:02.043867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:02.044005 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:02.044229 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:23:02.044851 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.046032 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.048407 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.049085 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.049219 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:02.050092 (+ 873us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:23:02.050853 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:23:02.050998 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:02.051866 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 15:23:02.051960 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 15:23:02.082172 16263 raft_consensus.cc:1261] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 1 FOLLOWER]: Refusing update from remote peer e99b233eb3224036ba6fa2ddd5a57983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:02.083487 16119 raft_consensus.cc:1261] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 1 FOLLOWER]: Refusing update from remote peer e99b233eb3224036ba6fa2ddd5a57983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:02.083509 16470 consensus_queue.cc:1035] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9a9952d52492eb703e23f2801a7a4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:02.085160 16470 consensus_queue.cc:1035] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36a2f9fa487446dac2a8cd4d8e6b47b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:02.127802 16479 mvcc.cc:205] Tried to move back new op lower bound from 7333980496189984768 to 7333980495836291072. Current Snapshot: MvccSnapshot[applied={T|T < 7333980496189984768}]
I20260927 15:23:02.145520 16478 mvcc.cc:205] Tried to move back new op lower bound from 7333980496189984768 to 7333980495836291072. Current Snapshot: MvccSnapshot[applied={T|T < 7333980496189984768}]
I20260927 15:23:02.174126 15967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:02.174436 15967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:02.179090 15967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea2bfba34c7e462a9693d391b2ad71dd: Sending DeleteTablet for 3 replicas of tablet bbc51d82ce9a4ac284d3c0a07bbc6326
I20260927 15:23:02.180598 16099 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:02 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:43058
I20260927 15:23:02.181155 16243 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:02 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:51630
I20260927 15:23:02.182214 16377 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc51d82ce9a4ac284d3c0a07bbc6326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:02 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:52118
I20260927 15:23:02.184602 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16004
I20260927 15:23:02.195282 16487 tablet_replica.cc:330] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: stopping tablet replica
I20260927 15:23:02.196300 16488 tablet_replica.cc:330] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: stopping tablet replica
I20260927 15:23:02.197727 16487 raft_consensus.cc:2229] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:02.198380 16488 raft_consensus.cc:2229] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:23:02.199244 16488 raft_consensus.cc:2258] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:02.199812 16487 raft_consensus.cc:2258] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:02.202596 16487 ts_tablet_manager.cc:1905] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:02.204748 16488 ts_tablet_manager.cc:1905] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:02.205385 16486 tablet_replica.cc:330] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: stopping tablet replica
I20260927 15:23:02.215399 16486 raft_consensus.cc:2229] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:02.222687 16487 ts_tablet_manager.cc:1918] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:02.223137 16486 raft_consensus.cc:2258] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:02.223816 16487 log.cc:1196] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/bbc51d82ce9a4ac284d3c0a07bbc6326
I20260927 15:23:02.225576 16486 ts_tablet_manager.cc:1905] T bbc51d82ce9a4ac284d3c0a07bbc6326 P d36a2f9fa487446dac2a8cd4d8e6b47b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:02.228040 16487 ts_tablet_manager.cc:1939] T bbc51d82ce9a4ac284d3c0a07bbc6326 P afb9a9952d52492eb703e23f2801a7a4: Deleting consensus metadata
I20260927 15:23:02.228315 16488 ts_tablet_manager.cc:1918] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:02.230656 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.186265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.2.246.65:46467)
0927 15:23:02.186598 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:02.186609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:02.202404 (+ 15795us) server_negotiation.cc:407] Connection header received
0927 15:23:02.202618 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:02.202625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:02.202750 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:02.202909 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:02.207973 (+ 5064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.209250 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.216187 (+ 6937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.217020 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.217256 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:02.221181 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:02.221206 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:02.221232 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:02.221299 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:02.226270 (+ 4971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.227368 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.227385 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.227389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.227630 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.228124 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.228129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.228133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.228751 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:02.229178 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:02.229430 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:23:02.230174 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:23:02.231604 16159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:02.232864 16488 log.cc:1196] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/bbc51d82ce9a4ac284d3c0a07bbc6326
I20260927 15:23:02.235782 15954 catalog_manager.cc:4841] TS afb9a9952d52492eb703e23f2801a7a4 (127.2.246.66:41803): tablet bbc51d82ce9a4ac284d3c0a07bbc6326 (table test-table [id=93cdab270fce4b21bc2df069437a7b8a]) successfully deleted
I20260927 15:23:02.236945 16488 ts_tablet_manager.cc:1939] T bbc51d82ce9a4ac284d3c0a07bbc6326 P e99b233eb3224036ba6fa2ddd5a57983: Deleting consensus metadata
I20260927 15:23:02.239683 15953 catalog_manager.cc:4841] TS e99b233eb3224036ba6fa2ddd5a57983 (127.2.246.67:45177): tablet bbc51d82ce9a4ac284d3c0a07bbc6326 (table test-table [id=93cdab270fce4b21bc2df069437a7b8a]) successfully deleted
I20260927 15:23:02.353615 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16004
I20260927 15:23:02.372895 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16168
W20260927 15:23:02.372789 15954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:46467 (error 108)
I20260927 15:23:02.397182 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.390146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52424)
0927 15:23:02.396766 (+ 6620us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.396940 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6424,"threads_started":1}
W20260927 15:23:02.397688 15954 catalog_manager.cc:4584] TS d36a2f9fa487446dac2a8cd4d8e6b47b (127.2.246.65:46467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbc51d82ce9a4ac284d3c0a07bbc6326: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
I20260927 15:23:02.399225 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.373699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.2.246.66:41803)
0927 15:23:02.373901 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:02.373908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:02.373961 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:23:02.374206 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:02.374216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:02.374340 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:02.374463 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:02.376890 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.378655 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.381557 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.382520 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.382685 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:02.388559 (+ 5874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:02.388591 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:02.388622 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:02.388693 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:02.393916 (+ 5223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.395402 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.395419 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.395425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.395673 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.396294 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.396303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.396309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.396843 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:02.397479 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:02.398428 (+ 949us) server_negotiation.cc:299] Negotiation successful
0927 15:23:02.398767 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:23:02.400053 16303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:02.451557 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.450973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52436)
0927 15:23:02.451257 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.451375 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 15:23:02.452898 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.452451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52446)
0927 15:23:02.452651 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.452738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 15:23:02.510994 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16168
I20260927 15:23:02.513471 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.512854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52454)
0927 15:23:02.513158 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.513266 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 15:23:02.515321 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.514790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52464)
0927 15:23:02.515037 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.515129 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 15:23:02.529812 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16312
I20260927 15:23:02.545794 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.530676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.2.246.67:45177)
0927 15:23:02.530862 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:02.530871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:02.530938 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:23:02.531163 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:02.531168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:02.531233 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:02.531342 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:02.532708 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.533589 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.534962 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:02.535430 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:02.535571 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:02.538693 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:02.538706 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:02.538722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:02.538769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:02.542654 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.543454 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.543463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.543467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.543656 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:02.544047 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:02.544052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:02.544055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:02.544449 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:02.544791 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:02.545079 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 15:23:02.545322 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:23:02.546991 16437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:02.596499 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.595729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52472)
0927 15:23:02.596068 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.596246 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 15:23:02.598286 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.597557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52476)
0927 15:23:02.597887 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.598060 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 15:23:02.671931 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16312
I20260927 15:23:02.692440 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 15934
I20260927 15:23:02.693930 15997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:02.864898 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:02.863353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:46467 (local address 127.0.0.1:52490)
0927 15:23:02.864462 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:02.864636 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688}
I20260927 15:23:02.865315 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 15934
I20260927 15:23:02.893231 3033 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:23:03.241307 3033 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:23:03.241555 3033 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:23:03.588896 3033 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:23:03.589 UTC [14986] LOG: received smart shutdown request
2026-09-27 15:23:03.593 UTC [14986] LOG: background worker "logical replication launcher" (PID 14995) exited with exit code 1
2026-09-27 15:23:03.593 UTC [14990] LOG: shutting down
2026-09-27 15:23:03.787 UTC [14986] LOG: database system is shut down
2026-09-27T15:23:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (62821 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:23:04 dist-test-slave-14jq krb5kdc[16511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:23:04 dist-test-slave-14jq krb5kdc[16511](info): set up 2 sockets
Sep 27 15:23:04 dist-test-slave-14jq krb5kdc[16511](info): commencing operation
krb5kdc: starting...
W20260927 15:23:05.090682 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:23:05 dist-test-slave-14jq krb5kdc[16511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:23:05.371338 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42103
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:05.713614 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:05.714049 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:05.714206 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:05.721985 16524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:05.722111 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:05.722188 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:05.722283 16524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:05.722352 16524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:05.733078 16524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:05.734591 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:05.735949 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:05.749181 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:05.749142 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:05.749231 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:05.751573 16524 server_base.cc:1034] running on GCE node
I20260927 15:23:05.752686 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:05.754637 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:05.755900 16524 hybrid_clock.cc:648] HybridClock initialized: now 1790522585755817 us; error 66 us; skew 500 ppm
I20260927 15:23:05.756423 16524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:05 dist-test-slave-14jq krb5kdc[16511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:05.760385 16524 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:05.762723 16524 webserver.cc:460] Webserver started at http://127.2.246.126:46471/ using document root <none> and password file <none>
I20260927 15:23:05.763739 16524 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:05.763876 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:05.764199 16524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:05.768090 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "049d6e00a15d41b89da1e27beb6ee141"
format_stamp: "Formatted at 2026-09-27 15:23:05 on dist-test-slave-14jq"
server_key: "be3ecff2f27c9f1bebc5ca35642ec5d7"
server_key_iv: "8d55b9b917e07f661d44848fb978d003"
server_key_version: "encryptionkey@0"
I20260927 15:23:05.769074 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "049d6e00a15d41b89da1e27beb6ee141"
format_stamp: "Formatted at 2026-09-27 15:23:05 on dist-test-slave-14jq"
server_key: "be3ecff2f27c9f1bebc5ca35642ec5d7"
server_key_iv: "8d55b9b917e07f661d44848fb978d003"
server_key_version: "encryptionkey@0"
I20260927 15:23:05.776332 16524 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:23:05.781175 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:05.783329 16524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:23:05.783628 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "049d6e00a15d41b89da1e27beb6ee141"
format_stamp: "Formatted at 2026-09-27 15:23:05 on dist-test-slave-14jq"
server_key: "be3ecff2f27c9f1bebc5ca35642ec5d7"
server_key_iv: "8d55b9b917e07f661d44848fb978d003"
server_key_version: "encryptionkey@0"
I20260927 15:23:05.783941 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:05.803344 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:05.809324 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:05.809758 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:05.834951 16524 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:42103
I20260927 15:23:05.835009 16590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:42103 every 8 connection(s)
I20260927 15:23:05.838920 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:05.841508 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16524
I20260927 15:23:05.841755 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:23:05.842674 3033 external_mini_cluster.cc:1428] Setting key 9414e5d8d856b531c1efe01f4e04effd
I20260927 15:23:05.845355 16591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:05.855305 16591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.125@KRBTEST.COM" created.
Sep 27 15:23:05 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:05.885932 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.859679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42103 (local address 127.0.0.1:45254)
0927 15:23:05.860994 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:05.861059 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:05.863985 (+ 2926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:05.865036 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:05.865044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:05.865976 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:05.866373 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:05.866393 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:05.868177 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:05.868185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:05.869408 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:05.869422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:05.870652 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:05.870660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:05.870879 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:05.871720 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:05.871753 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:05.875449 (+ 3696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:05.879630 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:05.879658 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:05.879682 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:05.880590 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:05.881274 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:05.881282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:05.881288 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:05.881617 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:05.883013 (+ 1396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:05.883022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:05.883256 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 15:23:05.883529 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 15:23:05.884485 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":285,"thread_start_us":570,"threads_started":1}
I20260927 15:23:05.885995 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.861744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.2.246.126:42103)
0927 15:23:05.862889 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:05.862899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:05.862979 (+ 80us) server_negotiation.cc:407] Connection header received
0927 15:23:05.864479 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:05.864486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:05.864778 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:05.865012 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:05.866596 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:05.868003 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:05.869580 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:05.870383 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:05.870624 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:05.875939 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:05.875950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:05.875966 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:05.876010 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:05.879347 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:05.880860 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:05.880869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:05.880872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:05.881090 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:05.881815 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:05.881820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:05.881822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:05.882762 (+ 940us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:05.883028 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:05.883753 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 15:23:05.884183 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":627,"threads_started":1}
I20260927 15:23:05.890406 16591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:23:05.892366 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.891661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58488)
0927 15:23:05.891951 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:05.892154 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 15:23:05.892807 16541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
I20260927 15:23:05.895259 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.894671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58494)
0927 15:23:05.894894 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:05.895015 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 15:23:05.895599 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:40907: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
I20260927 15:23:05.909646 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:40907
--webserver_interface=127.2.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:05.945065 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } attempt: 1
I20260927 15:23:05.946578 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.946066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58504)
0927 15:23:05.946305 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:05.946440 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 15:23:05.947745 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:05.947374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58506)
0927 15:23:05.947509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:05.947589 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:23:05.947930 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:40907: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
I20260927 15:23:06.016245 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } attempt: 2
I20260927 15:23:06.017762 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.016995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58518)
0927 15:23:06.017336 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.017504 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 15:23:06.019321 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.018675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58520)
0927 15:23:06.018915 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.019085 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 15:23:06.019645 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:40907: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
I20260927 15:23:06.111033 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } attempt: 3
I20260927 15:23:06.112749 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.111816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58528)
0927 15:23:06.112324 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.112527 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 15:23:06.114316 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.113705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58542)
0927 15:23:06.113944 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.114066 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:23:06.114624 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:40907: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
I20260927 15:23:06.258021 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } attempt: 4
I20260927 15:23:06.260365 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.258814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58552)
0927 15:23:06.259893 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.260083 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260927 15:23:06.262001 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.261418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58566)
0927 15:23:06.261624 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.261770 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 15:23:06.262324 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:40907: Network error: Client connection negotiation failed: client connection to 127.2.246.125:40907: connect: Connection refused (error 111)
W20260927 15:23:06.371132 16598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:06.371551 16598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:06.371732 16598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:06.380607 16598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:06.380774 16598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:06.380858 16598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:06.380990 16598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:06.381081 16598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:06.394178 16598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:40907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:06.396080 16598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:06.398002 16598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:06.415051 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:06.415402 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:06.416096 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:06.416179 16598 server_base.cc:1034] running on GCE node
I20260927 15:23:06.417089 16598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:06.418998 16598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:06.420320 16598 hybrid_clock.cc:648] HybridClock initialized: now 1790522586420228 us; error 63 us; skew 500 ppm
I20260927 15:23:06.420738 16598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:06 dist-test-slave-14jq krb5kdc[16511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:06.424605 16598 init.cc:377] Logged in from keytab as kudu/127.2.246.125@KRBTEST.COM (short username kudu)
I20260927 15:23:06.426800 16598 webserver.cc:460] Webserver started at http://127.2.246.125:32959/ using document root <none> and password file <none>
I20260927 15:23:06.427634 16598 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:06.427832 16598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:06.428165 16598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:06.431700 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data/instance:
uuid: "426dd61dd8f041448d515f847f484b6a"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "68d6c565d40b414db8ac2841c889c0fb"
server_key_iv: "b935a4717b48bc93196408d9446ad2f2"
server_key_version: "encryptionkey@0"
I20260927 15:23:06.432617 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal/instance:
uuid: "426dd61dd8f041448d515f847f484b6a"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "68d6c565d40b414db8ac2841c889c0fb"
server_key_iv: "b935a4717b48bc93196408d9446ad2f2"
server_key_version: "encryptionkey@0"
I20260927 15:23:06.439348 16598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:23:06.443982 16613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:06.446218 16598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:23:06.446439 16598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "426dd61dd8f041448d515f847f484b6a"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "68d6c565d40b414db8ac2841c889c0fb"
server_key_iv: "b935a4717b48bc93196408d9446ad2f2"
server_key_version: "encryptionkey@0"
I20260927 15:23:06.446666 16598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:06.472071 16598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:06.477178 16598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:06.477456 16598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:06.500715 16598 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.125:40907
I20260927 15:23:06.500838 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.125:40907 every 8 connection(s)
I20260927 15:23:06.502286 16598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
I20260927 15:23:06.507552 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:06.509884 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16598
I20260927 15:23:06.510107 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-1/wal/instance
I20260927 15:23:06.510574 3033 external_mini_cluster.cc:1428] Setting key 42fcef4ffe216b679286026be2a3ead1
I20260927 15:23:06.517198 16665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.124@KRBTEST.COM" created.
Sep 27 15:23:06 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:06.547550 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.520836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.2.246.126:42103)
0927 15:23:06.521506 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:06.521514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:06.522871 (+ 1357us) server_negotiation.cc:407] Connection header received
0927 15:23:06.525273 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:06.525284 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:06.525414 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:06.525585 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:06.527808 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.529028 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.531543 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.532010 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.532162 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.537431 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:06.537442 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:06.537451 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:06.537539 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:06.541415 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.543783 (+ 2368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.543791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.543799 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.543938 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.544814 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.544857 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.544860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.545380 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:06.545636 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:06.546742 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0927 15:23:06.547064 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":306,"threads_started":1}
I20260927 15:23:06.548925 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.521564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42103 (local address 127.0.0.1:45256)
0927 15:23:06.522620 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.522684 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:06.524876 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:06.525954 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:06.525978 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:06.526712 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:06.527501 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.527525 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.529313 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.529319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.531264 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.531285 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.532498 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.532515 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.532796 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.533772 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:06.533809 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:06.537045 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:06.541720 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.541740 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.541770 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.543530 (+ 1760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.544069 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.544074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.544078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.544545 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.545611 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:06.545623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:06.546273 (+ 650us) client_negotiation.cc:769] Sending connection context
0927 15:23:06.546626 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 15:23:06.547340 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":478,"threads_started":1}
I20260927 15:23:06.551467 16665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:23:06.561650 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } attempt: 5
Sep 27 15:23:06 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
I20260927 15:23:06.566111 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:39881
--webserver_interface=127.2.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:23:06 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:23:06.574021 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.552664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58580)
0927 15:23:06.552935 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.552957 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:06.553093 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:06.554739 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:06.554744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:06.555399 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:06.555599 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.555609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.557300 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.557309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.558445 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.558461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.559644 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.559650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.559901 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.560693 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:06.560716 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:06.564404 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:06.569805 (+ 5401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.569815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.569822 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.570828 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.571394 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.571398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.571401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.571624 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.572641 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:06.572647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:06.572843 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 15:23:06.573027 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:23:06.573423 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"spinlock_wait_cycles":40832}
I20260927 15:23:06.574321 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.553062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.2.246.125:40907)
0927 15:23:06.554090 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:06.554099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:06.554136 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:23:06.554211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:06.554218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:06.554512 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:06.554763 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:06.555803 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.557123 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.558635 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.559435 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.559591 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.564770 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:06.564794 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:06.564824 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:06.564892 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:06.569543 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.571023 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.571036 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.571042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.571239 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.571790 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.571799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.571806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.572467 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:06.572731 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:06.573262 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 15:23:06.573644 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":499,"threads_started":1}
I20260927 15:23:06.577953 16665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:23:06.579474 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.578832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53476)
0927 15:23:06.579127 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.579248 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260927 15:23:06.579980 16617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.582446 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.581745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53484)
0927 15:23:06.582078 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.582237 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260927 15:23:06.582937 16665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.583432 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.563076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.2.246.125:40907)
0927 15:23:06.564336 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:06.564343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:06.564373 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:23:06.564435 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:06.564439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:06.564523 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:06.565869 (+ 1346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:06.566204 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.567861 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.569890 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.570457 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.570634 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.575129 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:06.575139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:06.575147 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:06.575191 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:06.579677 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.580873 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.580880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.580883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.581091 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:06.581606 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:06.581612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:06.581615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:06.582091 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:06.582298 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:06.582752 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 15:23:06.583038 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":678,"threads_started":1}
I20260927 15:23:06.583869 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.562747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58586)
0927 15:23:06.563169 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.563196 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:06.563353 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:06.564760 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:06.564765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:06.565303 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:06.565573 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.565589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.568138 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.568147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.569689 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:06.569720 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:06.570669 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:06.570675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:06.570918 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:06.571815 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:06.571847 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:06.574837 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:06.579932 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.579941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.579949 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.580649 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.581254 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:06.581259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:06.581263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:06.581423 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:06.582240 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:06.582245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:06.582450 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 15:23:06.582645 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 15:23:06.583533 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260927 15:23:06.585089 16591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:23:06.586740 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.586243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53488)
0927 15:23:06.586464 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.586570 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 15:23:06.588376 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.587873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53500)
0927 15:23:06.588083 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.588172 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 15:23:06.588794 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.632431 16665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 1
I20260927 15:23:06.634109 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.633379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53504)
0927 15:23:06.633678 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.633885 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 15:23:06.635550 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.635126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53510)
0927 15:23:06.635322 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.635418 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:23:06.635856 16665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.640154 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 1
I20260927 15:23:06.641701 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.641049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53516)
0927 15:23:06.641344 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.641477 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 15:23:06.643018 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.642603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53528)
0927 15:23:06.642775 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.642873 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:23:06.643362 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.704232 16665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 2
I20260927 15:23:06.706084 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.705243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53544)
0927 15:23:06.705573 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.705746 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 15:23:06.707621 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.707149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53556)
0927 15:23:06.707356 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.707463 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 15:23:06.707935 16665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.711683 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 2
I20260927 15:23:06.713111 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.712529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53570)
0927 15:23:06.712802 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.712915 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 15:23:06.714407 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.713993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53572)
0927 15:23:06.714156 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.714259 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:23:06.714691 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.799211 16665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 3
I20260927 15:23:06.800951 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.800277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53588)
0927 15:23:06.800574 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.800723 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 15:23:06.802536 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.801886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53594)
0927 15:23:06.802240 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.802375 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260927 15:23:06.802831 16665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.821025 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 3
I20260927 15:23:06.822706 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.822035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53608)
0927 15:23:06.822261 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.822373 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 15:23:06.824074 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.823564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53616)
0927 15:23:06.823812 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.823925 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260927 15:23:06.824396 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
W20260927 15:23:06.939270 16673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:06.939604 16673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:06.939745 16673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:23:06.946161 16665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 4
I20260927 15:23:06.947705 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.947071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53620)
0927 15:23:06.947349 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.947501 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 15:23:06.948103 16673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:06.948252 16673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:06.948335 16673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:06.948433 16673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:06.948519 16673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:06.949290 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:06.948867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53632)
0927 15:23:06.949050 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:06.949160 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:23:06.949563 16665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:06.959394 16673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42103,127.2.246.125:40907,127.2.246.124:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:39881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:06.960762 16673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:06.962128 16673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:06.975996 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:06.976576 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:06.976189 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:06.978322 16673 server_base.cc:1034] running on GCE node
I20260927 15:23:06.979310 16673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:06.980926 16673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:06.982141 16673 hybrid_clock.cc:648] HybridClock initialized: now 1790522586982053 us; error 83 us; skew 500 ppm
I20260927 15:23:06.982538 16673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:06 dist-test-slave-14jq krb5kdc[16511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:06.986186 16673 init.cc:377] Logged in from keytab as kudu/127.2.246.124@KRBTEST.COM (short username kudu)
I20260927 15:23:06.987989 16673 webserver.cc:460] Webserver started at http://127.2.246.124:39673/ using document root <none> and password file <none>
I20260927 15:23:06.988711 16673 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:06.988858 16673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:06.989177 16673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:06.992201 16673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data/instance:
uuid: "077c83ad0c1c496dbc3d3efa2904e443"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "95a59364474e007827b8048468ca77c4"
server_key_iv: "85c6a4b093f5299373f9b77f8ecb4267"
server_key_version: "encryptionkey@0"
I20260927 15:23:06.993028 16673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal/instance:
uuid: "077c83ad0c1c496dbc3d3efa2904e443"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "95a59364474e007827b8048468ca77c4"
server_key_iv: "85c6a4b093f5299373f9b77f8ecb4267"
server_key_version: "encryptionkey@0"
I20260927 15:23:06.998706 16673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:23:07.001698 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 4
I20260927 15:23:07.003090 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:07.003240 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.002657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53642)
0927 15:23:07.002908 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.003016 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 15:23:07.004618 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.004161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53644)
0927 15:23:07.004356 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.004458 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 15:23:07.004865 16591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:39881: Network error: Client connection negotiation failed: client connection to 127.2.246.124:39881: connect: Connection refused (error 111)
I20260927 15:23:07.005030 16673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:07.005232 16673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "077c83ad0c1c496dbc3d3efa2904e443"
format_stamp: "Formatted at 2026-09-27 15:23:06 on dist-test-slave-14jq"
server_key: "95a59364474e007827b8048468ca77c4"
server_key_iv: "85c6a4b093f5299373f9b77f8ecb4267"
server_key_version: "encryptionkey@0"
I20260927 15:23:07.005453 16673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:07.029908 16673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:07.034402 16673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:07.034708 16673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:07.057636 16673 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.124:39881
I20260927 15:23:07.057747 16740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.124:39881 every 8 connection(s)
I20260927 15:23:07.059293 16673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
I20260927 15:23:07.064900 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:07.067513 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16673
I20260927 15:23:07.067715 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522351366233-3033-0/minicluster-data/master-2/wal/instance
I20260927 15:23:07.068217 3033 external_mini_cluster.cc:1428] Setting key bf8fb94e6d642a520d922eae42e05dee
I20260927 15:23:07.078157 16741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:07.101044 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.070628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.2.246.124:39881)
0927 15:23:07.071932 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.071955 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.072033 (+ 78us) server_negotiation.cc:407] Connection header received
0927 15:23:07.074198 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.074243 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.075031 (+ 788us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.075625 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.077124 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.079334 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.082133 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.084004 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.084270 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.089942 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.089954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.089970 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.090027 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.094511 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.095689 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.095702 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.095712 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.095942 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.096432 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.096437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.096440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.097735 (+ 1295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.098181 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.098677 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.099299 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":436,"threads_started":1}
I20260927 15:23:07.103185 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.082662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42103 (local address 127.0.0.1:45260)
0927 15:23:07.083640 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.083690 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:07.083886 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:07.084754 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:07.084762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:07.085609 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:07.086020 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.086032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.087099 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.087105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.088462 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.088479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.089926 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.089939 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.090270 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.091276 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:07.091297 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:07.094870 (+ 3573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:07.099750 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.099765 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.099772 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.100341 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.100857 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.100861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.100863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.101034 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.101967 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:07.101975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:07.102198 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 15:23:07.102415 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:23:07.102830 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":499,"threads_started":1}
I20260927 15:23:07.103451 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.082716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.2.246.126:42103)
0927 15:23:07.084368 (+ 1652us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.084373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.084389 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:23:07.084445 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.084449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.084522 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.084654 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.086192 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.086947 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.088728 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.089408 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.089686 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.095213 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.095221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.095226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.095292 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.099481 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.100505 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.100510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.100514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.100717 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.101172 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.101176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.101179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.101762 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.102007 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.102642 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.102979 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1380,"threads_started":1}
I20260927 15:23:07.105299 16741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:07.122555 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.106373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:40907 (local address 127.0.0.1:58598)
0927 15:23:07.106681 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.106704 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:07.106838 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:07.107487 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:07.107491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:07.107976 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:07.108144 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.108154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.109350 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.109358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.110558 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.110570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.111727 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.111732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.111897 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.112687 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:07.112704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:07.114920 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:07.119410 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.119419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.119427 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.119982 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.120565 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.120569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.120572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.120787 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.121651 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:07.121655 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:07.121815 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 15:23:07.122039 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 15:23:07.122292 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 15:23:07.123500 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.106427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.2.246.125:40907)
0927 15:23:07.107102 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.107112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.107136 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:23:07.107211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.107217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.107301 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.107452 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.108338 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.109208 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.110723 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.111488 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.111658 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.115265 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.115276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.115285 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.115378 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.119206 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.120172 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.120179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.120182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.120401 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.121031 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.121036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.121038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.121477 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.121678 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.122202 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.122853 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":344,"threads_started":1}
I20260927 15:23:07.124807 16741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:23:07.125990 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.105541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.2.246.126:42103)
0927 15:23:07.105754 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.105760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.105908 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:23:07.106218 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.106226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.106319 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.106488 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.108407 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.109293 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.110861 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.111302 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.111413 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.117860 (+ 6447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.117871 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.117877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.117970 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.121285 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.123048 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.123056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.123060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.123395 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.124070 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.124082 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.124085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.124460 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.124832 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.125281 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.125557 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:23:07.141314 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.126114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.2.246.124:39881)
0927 15:23:07.126294 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.126302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.126368 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:23:07.126588 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.126593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.126660 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.126768 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.127758 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.128634 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.129887 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.130339 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.130486 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.134694 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.134704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.134711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.134766 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.138470 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.139236 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.139244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.139250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.139373 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.139753 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.139760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.139764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.140099 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.140267 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.140630 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.140887 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":35}
I20260927 15:23:07.141400 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.125809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53660)
0927 15:23:07.126292 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.126316 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:07.126464 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:07.126783 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:07.126788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:07.127381 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:07.127606 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.127624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.128765 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.128773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.129710 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.129722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.130497 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.130502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.130794 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.131536 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:07.131560 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:07.134302 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:07.138671 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.138678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.138682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.139103 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.139494 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.139501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.139504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.139648 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.140219 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:07.140224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:07.140390 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 15:23:07.140534 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:23:07.141052 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":210}
I20260927 15:23:07.145573 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.128065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.2.246.125:40907)
0927 15:23:07.128275 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.128281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.128316 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:23:07.128387 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.128392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.128470 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.128580 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.129959 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.130709 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.132242 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.132709 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.132862 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.139032 (+ 6170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.139038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.139042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.139088 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.142102 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.143209 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.143216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.143220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.143407 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.143874 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.143879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.143882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.144204 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.144577 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.144883 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.145193 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:23:07.147538 16741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: Bootstrap starting.
I20260927 15:23:07.151678 16741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:07.152841 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:23:07.153059 16741 log.cc:824] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:07.157469 16741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: No bootstrap required, opened a new log
I20260927 15:23:07.170138 16741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.170920 16741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:07.171128 16741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c83ad0c1c496dbc3d3efa2904e443, State: Initialized, Role: FOLLOWER
I20260927 15:23:07.172144 16741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.180379 16756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.183164 16756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.183161 16741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:07.188000 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.161567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.2.246.126:42103)
0927 15:23:07.161748 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.161754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.162495 (+ 741us) server_negotiation.cc:407] Connection header received
0927 15:23:07.162807 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.162814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.162898 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.162970 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.166594 (+ 3624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.168230 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.171343 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.172391 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.172549 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.177382 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.177390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.177395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.177458 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.182367 (+ 4909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.184393 (+ 2026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.184403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.184408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.184621 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.185243 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.185249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.185252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.185703 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.186081 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.187112 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.187444 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:23:07.188117 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.161158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.2.246.124:39881)
0927 15:23:07.161388 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.161395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.166321 (+ 4926us) server_negotiation.cc:407] Connection header received
0927 15:23:07.166620 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.166627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.166723 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.166819 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.169524 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.170675 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.172995 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.173746 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.173921 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.179564 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.179575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.179583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.179650 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.183502 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.184758 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.184767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.184773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.184961 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.185599 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.185607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.185613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.186140 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.186469 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.187234 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.187589 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:23:07.194617 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.160766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.2.246.125:40907)
0927 15:23:07.160984 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.160991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.173226 (+ 12235us) server_negotiation.cc:407] Connection header received
0927 15:23:07.173414 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.173420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.173538 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.173669 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.176522 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.177681 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.180116 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.180995 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.181119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.185595 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.185605 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.185613 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.185670 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.190378 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.191532 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.191543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.191550 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.191777 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.192401 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.192410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.192419 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.193062 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.193358 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.193871 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.194171 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:23:07.203534 16767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:23:07.203720 16767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:23:07.199532 16673 master_runner.cc:186] Error getting master registration for 127.2.246.126:42103: OK, instance_id { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" instance_seqno: 1790522585827653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:23:07.211223 16673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:23:07.248827 16665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 5
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:23:07.266988 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.249577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53676)
0927 15:23:07.249975 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.249997 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:07.250129 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:07.251079 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:07.251087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:07.251687 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:07.251878 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.251887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.253095 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.253103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.254049 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.254062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.255058 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.255064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.255291 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.256060 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:07.256078 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:07.258342 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:07.262396 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.262402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.262406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.262908 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.263332 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.263337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.263341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.263476 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.264229 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:07.264237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:07.264515 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 15:23:07.265963 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0927 15:23:07.266467 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260927 15:23:07.268314 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.250405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.2.246.124:39881)
0927 15:23:07.250624 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.250630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.250649 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:23:07.250715 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.250719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.250814 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.250913 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.252045 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.252941 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.254218 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.254868 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.255007 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.258574 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.258580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.258583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.258619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.262214 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.263050 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.263054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.263057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.263204 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.263611 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.263620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.263622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.263998 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.264177 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.267515 (+ 3338us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.267859 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:23:07.277010 16665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: Bootstrap starting.
I20260927 15:23:07.282111 16591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } attempt: 5
I20260927 15:23:07.284211 16665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:07.286144 16665 log.cc:824] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: Log is configured to *not* fsync() on all Append() calls
W20260927 15:23:07.287549 16695 tablet.cc:2369] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:23:07 dist-test-slave-14jq krb5kdc[16511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:23:07.298914 16665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: No bootstrap required, opened a new log
I20260927 15:23:07.309230 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.282995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:39881 (local address 127.0.0.1:53686)
0927 15:23:07.283212 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:07.283238 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:07.283391 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:07.284270 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:07.284276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:07.284930 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:07.285197 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.285209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.287095 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.287105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.289285 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:07.289319 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.291207 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.291218 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:07.291539 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.292609 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:07.292635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:07.298929 (+ 6294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:07.305230 (+ 6301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.305239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.305244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.305981 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.306603 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:07.306609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:07.306613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:07.306808 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:07.307631 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:07.307637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:07.308003 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 15:23:07.308208 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 15:23:07.308691 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:23:07.313290 16665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.313920 16665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:07.314062 16665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426dd61dd8f041448d515f847f484b6a, State: Initialized, Role: FOLLOWER
I20260927 15:23:07.315058 16665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.316256 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:07.283519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.2.246.124:39881)
0927 15:23:07.283779 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:07.283789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:07.283815 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:23:07.283908 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:07.283916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:07.284027 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:07.285415 (+ 1388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:07.285633 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.286841 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.289761 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:07.290903 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:07.299198 (+ 8295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:07.299522 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:07.299530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:07.299539 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:07.299587 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:07.304899 (+ 5312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.306226 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.306232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.306236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.306433 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:07.306972 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:07.306977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:07.306981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:07.307460 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:07.310010 (+ 2550us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:07.310197 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 15:23:07.310552 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:23:07.317130 16770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.317588 16770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.319231 16665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:07.322914 16591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: Bootstrap starting.
I20260927 15:23:07.327378 16591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:07.328789 16591 log.cc:824] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:07.336082 16591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: No bootstrap required, opened a new log
I20260927 15:23:07.347853 16591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.348335 16591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:07.348462 16591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049d6e00a15d41b89da1e27beb6ee141, State: Initialized, Role: FOLLOWER
I20260927 15:23:07.349357 16591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.355940 16591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:07.357427 16776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.357935 16776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.424379 16792 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:23:07.439438 16794 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:23:07.429565 16524 master_runner.cc:186] Error getting master registration for 127.2.246.125:40907: OK, instance_id { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" instance_seqno: 1790522586494043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:23:07.440243 16524 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:23:07.450757 16756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:23:07.451193 16756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.453056 16756 leader_election.cc:288] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049d6e00a15d41b89da1e27beb6ee141 (127.2.246.126:42103), 426dd61dd8f041448d515f847f484b6a (127.2.246.125:40907)
I20260927 15:23:07.454638 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "077c83ad0c1c496dbc3d3efa2904e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426dd61dd8f041448d515f847f484b6a" is_pre_election: true
I20260927 15:23:07.455312 16640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c83ad0c1c496dbc3d3efa2904e443 in term 0.
I20260927 15:23:07.456110 16691 leader_election.cc:302] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c83ad0c1c496dbc3d3efa2904e443, 426dd61dd8f041448d515f847f484b6a; no voters:
I20260927 15:23:07.456707 16756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:23:07.457001 16756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:23:07.457144 16756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:07.457964 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "077c83ad0c1c496dbc3d3efa2904e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049d6e00a15d41b89da1e27beb6ee141" is_pre_election: true
I20260927 15:23:07.458565 16566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c83ad0c1c496dbc3d3efa2904e443 in term 0.
I20260927 15:23:07.462575 16756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
I20260927 15:23:07.463748 16756 leader_election.cc:288] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [CANDIDATE]: Term 1 election: Requested vote from peers 049d6e00a15d41b89da1e27beb6ee141 (127.2.246.126:42103), 426dd61dd8f041448d515f847f484b6a (127.2.246.125:40907)
I20260927 15:23:07.465663 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "077c83ad0c1c496dbc3d3efa2904e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426dd61dd8f041448d515f847f484b6a"
I20260927 15:23:07.465764 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "077c83ad0c1c496dbc3d3efa2904e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049d6e00a15d41b89da1e27beb6ee141"
I20260927 15:23:07.465971 16640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:07.466133 16566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:07.469637 16640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c83ad0c1c496dbc3d3efa2904e443 in term 1.
I20260927 15:23:07.470405 16691 leader_election.cc:302] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c83ad0c1c496dbc3d3efa2904e443, 426dd61dd8f041448d515f847f484b6a; no voters:
I20260927 15:23:07.470786 16598 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260927 15:23:07.471073 16598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:23:07.471388 16756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:07.470710 16566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c83ad0c1c496dbc3d3efa2904e443 in term 1.
I20260927 15:23:07.478246 16756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [term 1 LEADER]: Becoming Leader. State: Replica: 077c83ad0c1c496dbc3d3efa2904e443, State: Running, Role: LEADER
I20260927 15:23:07.479017 16756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } }
W20260927 15:23:07.481050 16619 tablet.cc:2369] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:07.491101 16795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077c83ad0c1c496dbc3d3efa2904e443. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.491671 16795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:07.494836 16797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:07.498267 16797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:07.526693 16640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [term 1 FOLLOWER]: Refusing update from remote peer 077c83ad0c1c496dbc3d3efa2904e443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:07.526692 16566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [term 1 FOLLOWER]: Refusing update from remote peer 077c83ad0c1c496dbc3d3efa2904e443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:07.527736 16795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:07.528189 16756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:07.551528 16776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077c83ad0c1c496dbc3d3efa2904e443. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.551908 16776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.553620 16770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077c83ad0c1c496dbc3d3efa2904e443. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.554030 16770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.555835 16798 mvcc.cc:205] Tried to move back new op lower bound from 7333980518497943552 to 7333980518319149056. Current Snapshot: MvccSnapshot[applied={T|T < 7333980518497943552}]
W20260927 15:23:07.564383 16545 tablet.cc:2369] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:07.568054 16800 mvcc.cc:205] Tried to move back new op lower bound from 7333980518497943552 to 7333980518319149056. Current Snapshot: MvccSnapshot[applied={T|T < 7333980518497943552}]
I20260927 15:23:07.568387 16799 mvcc.cc:205] Tried to move back new op lower bound from 7333980518497943552 to 7333980518319149056. Current Snapshot: MvccSnapshot[applied={T|T < 7333980518497943552}]
I20260927 15:23:07.570297 16776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.570680 16776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.582327 16770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.582626 16770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:23:07.584997 16795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.585342 16795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:07.585114 16803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "077c83ad0c1c496dbc3d3efa2904e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049d6e00a15d41b89da1e27beb6ee141" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42103 } } peers { permanent_uuid: "426dd61dd8f041448d515f847f484b6a" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 40907 } } peers { permanent_uuid: "077c83ad0c1c496dbc3d3efa2904e443" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 39881 } } }
I20260927 15:23:07.585496 16803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:07.588446 16797 catalog_manager.cc:1347] Generated new cluster ID: 2ced8667610d4aba94443a3f9cb7c285
I20260927 15:23:07.588613 16797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:23:07.608712 16797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:23:07.613073 16797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:23:07.630302 16797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077c83ad0c1c496dbc3d3efa2904e443: Generated new TSK 0
I20260927 15:23:07.631863 16797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:23:08.429346 16792 catalog_manager.cc:1259] Loaded cluster ID: 2ced8667610d4aba94443a3f9cb7c285
I20260927 15:23:08.429517 16792 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: loading cluster ID for follower catalog manager: success
I20260927 15:23:08.433063 16792 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: acquiring CA information for follower catalog manager: success
I20260927 15:23:08.436116 16792 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 049d6e00a15d41b89da1e27beb6ee141: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:23:08.446830 16794 catalog_manager.cc:1259] Loaded cluster ID: 2ced8667610d4aba94443a3f9cb7c285
I20260927 15:23:08.447032 16794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: loading cluster ID for follower catalog manager: success
I20260927 15:23:08.452560 16794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: acquiring CA information for follower catalog manager: success
I20260927 15:23:08.457441 16794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 426dd61dd8f041448d515f847f484b6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:23:08.742211 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16524
I20260927 15:23:08.743934 16586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:08.867230 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16524
I20260927 15:23:08.885715 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16598
I20260927 15:23:08.887292 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:08.998693 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16598
I20260927 15:23:09.024377 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16673
I20260927 15:23:09.025738 16736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:09.142333 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16673
2026-09-27T15:23:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5229 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:23:09 dist-test-slave-14jq krb5kdc[16818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:23:09 dist-test-slave-14jq krb5kdc[16818](info): set up 2 sockets
Sep 27 15:23:09 dist-test-slave-14jq krb5kdc[16818](info): commencing operation
krb5kdc: starting...
W20260927 15:23:10.306591 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:23:10 dist-test-slave-14jq krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:23:10.598963 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:45845
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:11.051151 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:11.051653 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:11.051855 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:11.060838 16831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:11.061095 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:11.061231 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:11.061342 16831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:11.061437 16831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:11.073458 16831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:11.076195 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:11.077752 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:11.095053 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:11.095060 16836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:11.095137 16839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:11.096249 16831 server_base.cc:1034] running on GCE node
I20260927 15:23:11.097349 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:11.099421 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:11.100690 16831 hybrid_clock.cc:648] HybridClock initialized: now 1790522591100634 us; error 45 us; skew 500 ppm
I20260927 15:23:11.101148 16831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:11 dist-test-slave-14jq krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:11.105130 16831 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:11.107290 16831 webserver.cc:460] Webserver started at http://127.2.246.126:41573/ using document root <none> and password file <none>
I20260927 15:23:11.108259 16831 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:11.108434 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:11.108834 16831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:11.112829 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d"
format_stamp: "Formatted at 2026-09-27 15:23:11 on dist-test-slave-14jq"
server_key: "d896d929664abfb0636d5b61340aa4e9"
server_key_iv: "e1fca1060a6bf94a5e0b2798c0d1a26b"
server_key_version: "encryptionkey@0"
I20260927 15:23:11.114001 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d"
format_stamp: "Formatted at 2026-09-27 15:23:11 on dist-test-slave-14jq"
server_key: "d896d929664abfb0636d5b61340aa4e9"
server_key_iv: "e1fca1060a6bf94a5e0b2798c0d1a26b"
server_key_version: "encryptionkey@0"
I20260927 15:23:11.121248 16831 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:23:11.126982 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:11.129323 16831 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 15:23:11.129658 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d"
format_stamp: "Formatted at 2026-09-27 15:23:11 on dist-test-slave-14jq"
server_key: "d896d929664abfb0636d5b61340aa4e9"
server_key_iv: "e1fca1060a6bf94a5e0b2798c0d1a26b"
server_key_version: "encryptionkey@0"
I20260927 15:23:11.130293 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:11.145421 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:11.152482 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:11.152980 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:11.179471 16831 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:45845
I20260927 15:23:11.179419 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:45845 every 8 connection(s)
I20260927 15:23:11.182022 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:11.187250 16898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:11.190124 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16831
I20260927 15:23:11.190392 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:23:11.191192 3033 external_mini_cluster.cc:1428] Setting key f2bcf3034c60959a4947714b1e208ec3
I20260927 15:23:11.203078 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d: Bootstrap starting.
I20260927 15:23:11.207228 16898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:11.208811 16898 log.cc:824] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:23:11 dist-test-slave-14jq krb5kdc[16818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:11.213826 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d: No bootstrap required, opened a new log
I20260927 15:23:11.223964 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:11.194121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.2.246.126:45845)
0927 15:23:11.195256 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:11.195280 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:11.195362 (+ 82us) server_negotiation.cc:407] Connection header received
0927 15:23:11.198160 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:11.198198 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:11.198875 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:11.199446 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:11.201321 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:11.203610 (+ 2289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:11.205243 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:11.206462 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:11.206682 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:11.212604 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:11.212620 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:11.212645 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:11.212716 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:11.217738 (+ 5022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:11.218928 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:11.218939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:11.218951 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:11.219237 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:11.219662 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:11.219667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:11.219670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:11.220602 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:11.220969 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:11.221390 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:23:11.222110 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":475,"threads_started":1}
I20260927 15:23:11.227804 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } }
I20260927 15:23:11.228473 16898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:11.228607 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbc7e7c9a7c4c1e9dbdfae26984624d, State: Initialized, Role: FOLLOWER
I20260927 15:23:11.229430 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } }
I20260927 15:23:11.229765 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:23:11.229786 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:23:11.229976 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:23:11.230192 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:11.233299 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } }
I20260927 15:23:11.234476 16898 leader_election.cc:302] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cbc7e7c9a7c4c1e9dbdfae26984624d; no voters:
I20260927 15:23:11.235700 16898 leader_election.cc:288] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:23:11.235939 16903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:11.238226 16903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [term 1 LEADER]: Becoming Leader. State: Replica: 7cbc7e7c9a7c4c1e9dbdfae26984624d, State: Running, Role: LEADER
I20260927 15:23:11.238560 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:11.238960 16903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } }
I20260927 15:23:11.243093 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } } }
I20260927 15:23:11.243043 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cbc7e7c9a7c4c1e9dbdfae26984624d. Latest consensus state: current_term: 1 leader_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbc7e7c9a7c4c1e9dbdfae26984624d" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 45845 } } }
I20260927 15:23:11.243649 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:11.243706 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:11.246121 16911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:11.252558 16911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:11.266678 16911 catalog_manager.cc:1347] Generated new cluster ID: b174dfda2f8c448e9bf6725c009d548e
I20260927 15:23:11.266862 16911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:23:11.278779 16911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:23:11.282866 16911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:23:11.292691 16911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cbc7e7c9a7c4c1e9dbdfae26984624d: Generated new TSK 0
I20260927 15:23:11.293815 16911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:23:11.346469 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16831
I20260927 15:23:11.348438 16893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:11.461125 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16831
2026-09-27T15:23:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2316 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:23:11 dist-test-slave-14jq krb5kdc[16926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:23:11 dist-test-slave-14jq krb5kdc[16926](info): set up 2 sockets
Sep 27 15:23:11 dist-test-slave-14jq krb5kdc[16926](info): commencing operation
krb5kdc: starting...
W20260927 15:23:12.615012 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:23:12 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:23:12.909988 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:13.268504 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:13.268878 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:13.269033 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:13.276907 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:13.277027 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:13.277074 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:13.277191 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:13.277266 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:13.287892 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:13.289343 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:13.290789 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:13.305122 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:13.304975 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:13.305065 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:13.306325 16939 server_base.cc:1034] running on GCE node
I20260927 15:23:13.307309 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:13.309131 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:13.310518 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790522593310426 us; error 79 us; skew 500 ppm
I20260927 15:23:13.311023 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:13 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:13.315059 16939 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:13.317334 16939 webserver.cc:460] Webserver started at http://127.2.246.126:33687/ using document root <none> and password file <none>
I20260927 15:23:13.318274 16939 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:13.318444 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:13.318819 16939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:13.322487 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "89b1dcd424084bf394170ecd3cd6a6c9"
format_stamp: "Formatted at 2026-09-27 15:23:13 on dist-test-slave-14jq"
server_key: "89df0c0c03e1fba87dca4276aff79573"
server_key_iv: "926f978bcd8551f92ea1c69a7cd32691"
server_key_version: "encryptionkey@0"
I20260927 15:23:13.323428 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "89b1dcd424084bf394170ecd3cd6a6c9"
format_stamp: "Formatted at 2026-09-27 15:23:13 on dist-test-slave-14jq"
server_key: "89df0c0c03e1fba87dca4276aff79573"
server_key_iv: "926f978bcd8551f92ea1c69a7cd32691"
server_key_version: "encryptionkey@0"
I20260927 15:23:13.330106 16939 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:23:13.334820 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:13.336828 16939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:13.337121 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "89b1dcd424084bf394170ecd3cd6a6c9"
format_stamp: "Formatted at 2026-09-27 15:23:13 on dist-test-slave-14jq"
server_key: "89df0c0c03e1fba87dca4276aff79573"
server_key_iv: "926f978bcd8551f92ea1c69a7cd32691"
server_key_version: "encryptionkey@0"
I20260927 15:23:13.337355 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:13.354395 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:13.360473 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:13.360904 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:13.387897 16939 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33997
I20260927 15:23:13.387952 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33997 every 8 connection(s)
I20260927 15:23:13.389603 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:13.390250 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 16939
I20260927 15:23:13.390528 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:23:13.391500 3033 external_mini_cluster.cc:1428] Setting key a3f5262629cbd18257e0685c85ddbf59
I20260927 15:23:13.395536 17006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:13.411120 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap starting.
Sep 27 15:23:13 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:13.415187 17006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:13.416705 17006 log.cc:824] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:13.421236 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: No bootstrap required, opened a new log
I20260927 15:23:13.425643 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:13.394428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.2.246.126:33997)
0927 15:23:13.395529 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:13.395562 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:13.395632 (+ 70us) server_negotiation.cc:407] Connection header received
0927 15:23:13.398120 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:13.398155 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:13.398762 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:13.399298 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:13.402336 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:13.404616 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:13.406633 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:13.407914 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:13.408187 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:13.414727 (+ 6540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:13.414741 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:13.414764 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:13.414828 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:13.419680 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:13.420930 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:13.420938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:13.420948 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:13.421198 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:13.421882 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:13.421888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:13.421891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:13.422684 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:13.423023 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:13.423607 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 15:23:13.424135 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":455,"threads_started":1}
I20260927 15:23:13.431711 17006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:13.432143 17006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:13.432210 17006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Initialized, Role: FOLLOWER
I20260927 15:23:13.432848 17006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:13.433084 17006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:23:13.433198 17006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:23:13.433336 17006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:13.435739 17006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:13.436408 17006 leader_election.cc:302] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b1dcd424084bf394170ecd3cd6a6c9; no voters:
I20260927 15:23:13.437222 17006 leader_election.cc:288] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:23:13.437476 17011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:13.438503 17011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Running, Role: LEADER
I20260927 15:23:13.439276 17011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:13.439721 17006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:13.443209 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b1dcd424084bf394170ecd3cd6a6c9. Latest consensus state: current_term: 1 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:13.444226 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:13.445055 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:13.445946 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:13.449052 17021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:13.454875 17021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:13.466934 17021 catalog_manager.cc:1347] Generated new cluster ID: 6fa88ba46b1a442882bc2c63609bc0b9
I20260927 15:23:13.467089 17021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:23:13.482810 17021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:23:13.485355 17021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:23:13.496575 17021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Generated new TSK 0
I20260927 15:23:13.498083 17021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:23:13.570022 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:14.002501 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:14.002945 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:14.003134 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:14.011556 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:14.011763 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:14.011965 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:23:14.024590 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:14.026516 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:14.028182 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:14.044744 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:14.044806 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:14.045167 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:14.046216 17032 server_base.cc:1034] running on GCE node
I20260927 15:23:14.047005 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:14.048247 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:14.049480 17032 hybrid_clock.cc:648] HybridClock initialized: now 1790522594049420 us; error 48 us; skew 500 ppm
I20260927 15:23:14.050031 17032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:14 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:14.054039 17032 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:23:14.056146 17032 webserver.cc:460] Webserver started at http://127.2.246.65:46547/ using document root <none> and password file <none>
I20260927 15:23:14.057099 17032 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:14.057263 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:14.057631 17032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:14.061568 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "c8f2265220d7413fb3092f0b8364db10"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "e3650af3ea02db239d2f2ce90a729ac7"
server_key_iv: "aa68449e16fd87667e42f3355bbecdac"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.062597 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "c8f2265220d7413fb3092f0b8364db10"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "e3650af3ea02db239d2f2ce90a729ac7"
server_key_iv: "aa68449e16fd87667e42f3355bbecdac"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.069368 17032 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 15:23:14.074503 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.076611 17032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:23:14.076903 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c8f2265220d7413fb3092f0b8364db10"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "e3650af3ea02db239d2f2ce90a729ac7"
server_key_iv: "aa68449e16fd87667e42f3355bbecdac"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.077198 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:14.117280 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:14.123886 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:14.124315 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:14.125662 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:14.127660 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:14.127784 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.127980 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:14.128060 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.168576 17032 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:36205
I20260927 15:23:14.168640 17159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:36205 every 8 connection(s)
I20260927 15:23:14.170353 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:23:14.174479 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17032
I20260927 15:23:14.174721 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:23:14.175586 3033 external_mini_cluster.cc:1428] Setting key c94f20d9c028f109b70506c32058b0ed
Sep 27 15:23:14 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:14.197336 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:14.174085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:55717 (local address 127.2.246.126:33997)
0927 15:23:14.174896 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:14.174906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:14.176376 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 15:23:14.179108 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:14.179113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:14.179205 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:14.179357 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:14.182022 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.182875 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.185817 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.186409 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.186580 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:14.190754 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:14.190761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:14.190764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:14.190808 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:14.193914 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:14.194758 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:14.194766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:14.194770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:14.194903 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:14.195377 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:14.195384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:14.195391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:14.195723 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:14.195884 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:14.196679 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 15:23:14.196945 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":380,"threads_started":1}
I20260927 15:23:14.198479 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:14.174896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:55717)
0927 15:23:14.176127 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:14.176203 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:14.178688 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:14.179715 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:14.179731 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:14.180614 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:14.181568 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:14.181598 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.183097 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.183106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:14.185530 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:14.185551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.186740 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.186751 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:14.187149 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:14.188054 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:14.188083 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:14.190442 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:14.194097 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:14.194107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:14.194128 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:14.194600 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:14.195028 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:14.195033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:14.195037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:14.195238 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:14.195871 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:14.195880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:14.196272 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 15:23:14.196579 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 15:23:14.197167 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":455,"threads_started":1}
I20260927 15:23:14.200608 17160 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:14.200982 17160 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:14.201820 17160 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
I20260927 15:23:14.204644 16971 ts_manager.cc:194] Registered new tserver with Master: c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205)
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:23:14.206511 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:55717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:23:14.246894 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:14.707508 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:14.707862 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:14.708048 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:14.716634 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:14.716781 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:14.716986 17166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:23:14.728055 17166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:14.729578 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:14.731218 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:14.748972 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:14.748919 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:14.749013 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:14.750651 17166 server_base.cc:1034] running on GCE node
I20260927 15:23:14.751374 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:14.752545 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:14.753930 17166 hybrid_clock.cc:648] HybridClock initialized: now 1790522594753793 us; error 71 us; skew 500 ppm
I20260927 15:23:14.754387 17166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:14 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:14.763957 17166 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:23:14.766003 17166 webserver.cc:460] Webserver started at http://127.2.246.66:34693/ using document root <none> and password file <none>
I20260927 15:23:14.766840 17166 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:14.767000 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:14.767345 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:14.770591 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "f079366c01476c87216b53a7b6bb0fee"
server_key_iv: "8a2dbf1c99ce4e6c2f9a646141877b33"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.771421 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "f079366c01476c87216b53a7b6bb0fee"
server_key_iv: "8a2dbf1c99ce4e6c2f9a646141877b33"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.777967 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:23:14.782862 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.785053 17166 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 15:23:14.785310 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "f079366c01476c87216b53a7b6bb0fee"
server_key_iv: "8a2dbf1c99ce4e6c2f9a646141877b33"
server_key_version: "encryptionkey@0"
I20260927 15:23:14.785566 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:14.805125 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:14.810022 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:14.810385 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:14.811615 17166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:14.813246 17166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:14.813357 17166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.813510 17166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:14.813583 17166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:14.856364 17166 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:40409
I20260927 15:23:14.856485 17293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:40409 every 8 connection(s)
I20260927 15:23:14.858274 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:23:14.862864 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17166
I20260927 15:23:14.863097 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:23:14.863662 3033 external_mini_cluster.cc:1428] Setting key da531c462b6d46ad0b41798d9c9125c4
Sep 27 15:23:14 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:14.887079 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:14.861882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:48569 (local address 127.2.246.126:33997)
0927 15:23:14.862500 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:14.862506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:14.863731 (+ 1225us) server_negotiation.cc:407] Connection header received
0927 15:23:14.866339 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:14.866345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:14.866458 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:14.866602 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:14.868632 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.869453 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.871641 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.872232 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.872416 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:14.878153 (+ 5737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:14.878166 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:14.878175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:14.878245 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:14.882650 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:14.884100 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:14.884110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:14.884116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:14.884343 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:14.884975 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:14.884981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:14.884984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:14.885355 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:14.885544 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:14.886382 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 15:23:14.886697 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":328,"threads_started":1}
I20260927 15:23:14.888203 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:14.862474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:48569)
0927 15:23:14.863495 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:14.863554 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:14.865999 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:14.866922 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:14.866934 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:14.867650 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:14.868400 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:14.868438 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.869637 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.869642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:14.871424 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:14.871442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:14.872539 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:14.872548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:14.873014 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:14.874111 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:14.874142 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:14.877653 (+ 3511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:14.883099 (+ 5446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:14.883164 (+ 65us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:14.883183 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:14.883888 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:14.884493 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:14.884497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:14.884500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:14.884757 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:14.885504 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:14.885512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:14.885975 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 15:23:14.886270 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 15:23:14.886877 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":411,"threads_started":1}
I20260927 15:23:14.890196 17294 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:14.890854 17294 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:14.892202 17294 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:23:14.894726 16971 ts_manager.cc:194] Registered new tserver with Master: c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409)
I20260927 15:23:14.895745 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:48569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:23:14.932559 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:15.210366 17160 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
W20260927 15:23:15.375239 17300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:15.375674 17300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:15.375877 17300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:15.385905 17300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:15.386147 17300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:15.386399 17300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:23:15.402591 17300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:15.404486 17300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:15.406314 17300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:15.424105 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:15.424844 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:15.424873 17308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:15.427603 17300 server_base.cc:1034] running on GCE node
I20260927 15:23:15.428474 17300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:15.429927 17300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:15.431232 17300 hybrid_clock.cc:648] HybridClock initialized: now 1790522595431135 us; error 76 us; skew 500 ppm
I20260927 15:23:15.431715 17300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:15 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:15.435896 17300 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:23:15.438217 17300 webserver.cc:460] Webserver started at http://127.2.246.67:43071/ using document root <none> and password file <none>
I20260927 15:23:15.439239 17300 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:15.439424 17300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:15.439829 17300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:15.444386 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "cf4f3b83fcee46758423d474d41defaa"
format_stamp: "Formatted at 2026-09-27 15:23:15 on dist-test-slave-14jq"
server_key: "c14dbcb32dcbb03035aba0f92bd1817f"
server_key_iv: "4b66ee6423dad09aba4bdd6929675664"
server_key_version: "encryptionkey@0"
I20260927 15:23:15.445544 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "cf4f3b83fcee46758423d474d41defaa"
format_stamp: "Formatted at 2026-09-27 15:23:15 on dist-test-slave-14jq"
server_key: "c14dbcb32dcbb03035aba0f92bd1817f"
server_key_iv: "4b66ee6423dad09aba4bdd6929675664"
server_key_version: "encryptionkey@0"
I20260927 15:23:15.453178 17300 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260927 15:23:15.459244 17315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:15.462107 17300 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.003s
I20260927 15:23:15.462430 17300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "cf4f3b83fcee46758423d474d41defaa"
format_stamp: "Formatted at 2026-09-27 15:23:15 on dist-test-slave-14jq"
server_key: "c14dbcb32dcbb03035aba0f92bd1817f"
server_key_iv: "4b66ee6423dad09aba4bdd6929675664"
server_key_version: "encryptionkey@0"
I20260927 15:23:15.462754 17300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:15.506574 17300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:15.513072 17300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:15.513521 17300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:15.515093 17300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:15.517004 17300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:15.517146 17300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:15.517325 17300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:15.517400 17300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:15.562799 17300 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:42241
I20260927 15:23:15.562865 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:42241 every 8 connection(s)
I20260927 15:23:15.564704 17300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:23:15.568120 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17300
I20260927 15:23:15.568447 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:23:15.569042 3033 external_mini_cluster.cc:1428] Setting key eb67969907e19a1a1f818ad301fbab55
Sep 27 15:23:15 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:15.591825 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.568506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:42255 (local address 127.2.246.126:33997)
0927 15:23:15.569225 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.569232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.570593 (+ 1361us) server_negotiation.cc:407] Connection header received
0927 15:23:15.573266 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.573272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.573397 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.573557 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:15.575444 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.576485 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.578549 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.579109 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.579271 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.583311 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:15.583319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:15.583324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:15.583380 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:15.587558 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.588661 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.588667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.588671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.588828 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.589641 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.589646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.589648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.590062 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:15.590224 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.591337 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.591534 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":325,"threads_started":1}
I20260927 15:23:15.593238 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.569334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:42255)
0927 15:23:15.570364 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.570428 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.572826 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.573809 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.573818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.574508 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:15.575189 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.575210 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.576718 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.576723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.578296 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.578312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.579406 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.579411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.579650 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.580524 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:15.580549 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:15.582921 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:15.587782 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:15.587795 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:15.587835 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:15.588474 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:15.589014 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:15.589019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:15.589023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:15.589415 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:15.590201 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:15.590209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:15.590744 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.591200 (+ 456us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.591768 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":402,"threads_started":1}
I20260927 15:23:15.595305 17428 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:15.595746 17428 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:15.596704 17428 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
I20260927 15:23:15.598888 16971 ts_manager.cc:194] Registered new tserver with Master: cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241)
I20260927 15:23:15.600035 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:42255
I20260927 15:23:15.600804 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:23:15.621729 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.606409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.2.246.126:33997)
0927 15:23:15.606644 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.606648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.607106 (+ 458us) server_negotiation.cc:407] Connection header received
0927 15:23:15.607246 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.607250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.607324 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.607444 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:15.609057 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.609954 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.611509 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.611920 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.612044 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.615209 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:15.615215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:15.615219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:15.615265 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:15.618674 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.619751 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.619755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.619758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.619914 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.620295 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.620298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.620300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.620654 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:15.620902 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.621247 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.621441 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:23:15.627146 16971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:23:15.636736 16971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:23:15.680770 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.655065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.2.246.65:36205)
0927 15:23:15.669619 (+ 14554us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.669635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.669687 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:23:15.670534 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.670601 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.671079 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.672533 (+ 1454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:15.672862 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.674694 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.677509 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.679268 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.679419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.679695 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.680235 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.680426 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14294,"thread_start_us":14091,"threads_started":1}
I20260927 15:23:15.681226 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.654776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:36205 (local address 127.0.0.1:53340)
0927 15:23:15.664203 (+ 9427us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.664303 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.664671 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.671515 (+ 6844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.671529 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.671597 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:15.672239 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.672257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.674851 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.674860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.677339 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.677353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.679517 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.679525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.679901 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.679982 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.680162 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.680427 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8815,"mutex_wait_us":55,"thread_start_us":19688,"threads_started":1}
I20260927 15:23:15.682884 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.655247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:42241 (local address 127.0.0.1:46562)
0927 15:23:15.670609 (+ 15362us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.670643 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.670904 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.674762 (+ 3858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.674768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.674798 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:15.675454 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.675472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.677562 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.677569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.679787 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.679800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.681880 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.681889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.682110 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.682153 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.682247 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.682417 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15127,"mutex_wait_us":8,"thread_start_us":14911,"threads_started":1}
I20260927 15:23:15.684260 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.656212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.2.246.67:42241)
0927 15:23:15.673867 (+ 17655us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.673881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.673932 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:23:15.674068 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.674077 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.674434 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.675343 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:15.675667 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.677373 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.679990 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.681508 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.681714 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.681991 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.683578 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.683834 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17397,"thread_start_us":17164,"threads_started":1}
I20260927 15:23:15.686955 17095 tablet_service.cc:1467] Processing CreateTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 (DEFAULT_TABLE table=test-table [id=afcede8e8350420a8a2e4ffe164326f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:15.688200 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.656269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.2.246.66:40409)
0927 15:23:15.674273 (+ 18004us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.674281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.674983 (+ 702us) server_negotiation.cc:407] Connection header received
0927 15:23:15.675367 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.675377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.675696 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.675931 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:15.677064 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.678912 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.682207 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.686984 (+ 4777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.687191 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.687395 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.687684 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.687852 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17826,"thread_start_us":17640,"threads_started":1}
I20260927 15:23:15.688238 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.655576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40409 (local address 127.0.0.1:45232)
0927 15:23:15.674847 (+ 19271us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.674876 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.675121 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.675950 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.675955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.675982 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:15.676513 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.676526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.679079 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.679086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.681926 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.681949 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.687259 (+ 5310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.687267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.687471 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.687499 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.687624 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.687792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19093,"mutex_wait_us":31,"thread_start_us":9797,"threads_started":1}
I20260927 15:23:15.688186 17363 tablet_service.cc:1467] Processing CreateTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 (DEFAULT_TABLE table=test-table [id=afcede8e8350420a8a2e4ffe164326f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:15.689240 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f6c0db4b5547ee8ab406f33114e261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:15.689664 17363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f6c0db4b5547ee8ab406f33114e261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:15.692335 17229 tablet_service.cc:1467] Processing CreateTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 (DEFAULT_TABLE table=test-table [id=afcede8e8350420a8a2e4ffe164326f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:15.693608 17229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f6c0db4b5547ee8ab406f33114e261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:15.710861 17449 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Bootstrap starting.
I20260927 15:23:15.713577 17447 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Bootstrap starting.
I20260927 15:23:15.714767 17448 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Bootstrap starting.
I20260927 15:23:15.715586 17449 tablet_bootstrap.cc:654] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:15.717132 17447 tablet_bootstrap.cc:654] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:15.717290 17449 log.cc:824] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:15.718734 17447 log.cc:824] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:15.719175 17448 tablet_bootstrap.cc:654] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:15.721310 17448 log.cc:824] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:15.722244 17449 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: No bootstrap required, opened a new log
I20260927 15:23:15.722571 17449 ts_tablet_manager.cc:1397] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 15:23:15.724965 17447 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: No bootstrap required, opened a new log
I20260927 15:23:15.725294 17447 ts_tablet_manager.cc:1397] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260927 15:23:15.725929 17448 tablet_bootstrap.cc:492] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: No bootstrap required, opened a new log
I20260927 15:23:15.726323 17448 ts_tablet_manager.cc:1397] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 15:23:15.734066 17449 raft_consensus.cc:348] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.734635 17449 raft_consensus.cc:374] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:15.734797 17449 raft_consensus.cc:729] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4932dedb5234a3b9d6ee1cf295889cc, State: Initialized, Role: FOLLOWER
I20260927 15:23:15.735603 17449 consensus_queue.cc:260] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.735783 17448 raft_consensus.cc:348] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.736583 17448 raft_consensus.cc:374] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:15.736390 17447 raft_consensus.cc:348] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.736758 17448 raft_consensus.cc:729] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f2265220d7413fb3092f0b8364db10, State: Initialized, Role: FOLLOWER
I20260927 15:23:15.737048 17447 raft_consensus.cc:374] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:15.737178 17447 raft_consensus.cc:729] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4f3b83fcee46758423d474d41defaa, State: Initialized, Role: FOLLOWER
I20260927 15:23:15.737226 17294 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
I20260927 15:23:15.737884 17449 ts_tablet_manager.cc:1428] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 15:23:15.737960 17447 consensus_queue.cc:260] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.738889 17448 consensus_queue.cc:260] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.739619 17428 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
I20260927 15:23:15.740693 17447 ts_tablet_manager.cc:1428] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260927 15:23:15.742587 17448 ts_tablet_manager.cc:1428] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 15:23:15.757028 17455 raft_consensus.cc:482] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:23:15.757362 17455 raft_consensus.cc:504] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.759364 17455 leader_election.cc:288] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241), c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409)
I20260927 15:23:15.769008 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.760012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:38379 (local address 127.2.246.67:42241)
0927 15:23:15.760459 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.760469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.761084 (+ 615us) server_negotiation.cc:407] Connection header received
0927 15:23:15.761408 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.761413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.761489 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.761606 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:15.762544 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.763373 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.766528 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.767666 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.767876 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.767960 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.768522 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.768695 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260927 15:23:15.769258 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.759833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:42241 (local address 127.2.246.65:38379)
0927 15:23:15.760908 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.760934 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.761223 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.761642 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.761648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.761684 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:15.762371 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.762385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.763547 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.763554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.766280 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.766302 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.768052 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.768059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.768291 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.768325 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.768447 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.768630 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":144,"thread_start_us":523,"threads_started":1}
I20260927 15:23:15.770051 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.759897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:34381 (local address 127.2.246.66:40409)
0927 15:23:15.760185 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.760193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.761241 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 15:23:15.761448 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.761453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.761517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.761636 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:15.762536 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.763285 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.767441 (+ 4156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.768592 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.768761 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.768836 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.769593 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.769730 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:23:15.770221 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.759959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40409 (local address 127.2.246.65:34381)
0927 15:23:15.761081 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:15.761103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:15.761269 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:15.761647 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:15.761652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:15.761763 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:15.762372 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.762384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.763427 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.763433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.766280 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:15.766302 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.769067 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.769129 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:15.769381 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.769423 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:23:15.769568 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:23:15.769741 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":164,"spinlock_wait_cycles":27520,"thread_start_us":752,"threads_started":1}
I20260927 15:23:15.770808 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f6c0db4b5547ee8ab406f33114e261" candidate_uuid: "c8f2265220d7413fb3092f0b8364db10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4f3b83fcee46758423d474d41defaa" is_pre_election: true
I20260927 15:23:15.771591 17383 raft_consensus.cc:2454] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f2265220d7413fb3092f0b8364db10 in term 0.
I20260927 15:23:15.771492 17249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f6c0db4b5547ee8ab406f33114e261" candidate_uuid: "c8f2265220d7413fb3092f0b8364db10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" is_pre_election: true
I20260927 15:23:15.772090 17249 raft_consensus.cc:2454] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f2265220d7413fb3092f0b8364db10 in term 0.
I20260927 15:23:15.772539 17049 leader_election.cc:302] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f2265220d7413fb3092f0b8364db10, cf4f3b83fcee46758423d474d41defaa; no voters:
I20260927 15:23:15.773069 17455 raft_consensus.cc:2781] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:23:15.773255 17455 raft_consensus.cc:482] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:23:15.773368 17455 raft_consensus.cc:3037] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:15.776057 17455 raft_consensus.cc:504] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.776847 17455 leader_election.cc:288] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [CANDIDATE]: Term 1 election: Requested vote from peers cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241), c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409)
I20260927 15:23:15.777616 17249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f6c0db4b5547ee8ab406f33114e261" candidate_uuid: "c8f2265220d7413fb3092f0b8364db10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
I20260927 15:23:15.777668 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f6c0db4b5547ee8ab406f33114e261" candidate_uuid: "c8f2265220d7413fb3092f0b8364db10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4f3b83fcee46758423d474d41defaa"
I20260927 15:23:15.778120 17249 raft_consensus.cc:3037] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:15.778180 17383 raft_consensus.cc:3037] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:15.780774 17249 raft_consensus.cc:2454] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f2265220d7413fb3092f0b8364db10 in term 1.
I20260927 15:23:15.781266 17383 raft_consensus.cc:2454] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f2265220d7413fb3092f0b8364db10 in term 1.
I20260927 15:23:15.781536 17048 leader_election.cc:302] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4932dedb5234a3b9d6ee1cf295889cc, c8f2265220d7413fb3092f0b8364db10; no voters:
I20260927 15:23:15.782073 17455 raft_consensus.cc:2781] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:15.782671 17455 raft_consensus.cc:686] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 1 LEADER]: Becoming Leader. State: Replica: c8f2265220d7413fb3092f0b8364db10, State: Running, Role: LEADER
I20260927 15:23:15.783293 17455 consensus_queue.cc:237] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } }
I20260927 15:23:15.788897 16971 catalog_manager.cc:5495] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f2265220d7413fb3092f0b8364db10 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "c8f2265220d7413fb3092f0b8364db10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260927 15:23:15.810077 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.803208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.2.246.65:36205)
0927 15:23:15.803428 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.803434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.803456 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:23:15.803581 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.803600 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.803713 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.803890 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:23:15.804450 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.805265 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.806910 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.807421 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.807566 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.808244 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:23:15.808985 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:23:15.809147 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.809457 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.809569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 15:23:15.817152 17429 tablet.cc:2369] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:15.821547 17383 raft_consensus.cc:1261] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Refusing update from remote peer c8f2265220d7413fb3092f0b8364db10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:15.821547 17249 raft_consensus.cc:1261] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Refusing update from remote peer c8f2265220d7413fb3092f0b8364db10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:15.823335 17455 consensus_queue.cc:1035] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:15.823746 17458 consensus_queue.cc:1035] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:15.863065 16969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:15.863520 16969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:15.869122 16969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Sending DeleteTablet for 3 replicas of tablet 48f6c0db4b5547ee8ab406f33114e261
I20260927 15:23:15.871030 17095 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:15 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:53340
I20260927 15:23:15.872046 17363 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:15 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:46562
I20260927 15:23:15.872295 17471 tablet_replica.cc:330] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: stopping tablet replica
I20260927 15:23:15.872938 17471 raft_consensus.cc:2229] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:23:15.874089 17471 raft_consensus.cc:2258] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:15.874891 17229 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f6c0db4b5547ee8ab406f33114e261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:15 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:45232
I20260927 15:23:15.876573 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17032
I20260927 15:23:15.883869 17472 tablet_replica.cc:330] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: stopping tablet replica
I20260927 15:23:15.884795 17472 raft_consensus.cc:2229] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:15.885733 17472 raft_consensus.cc:2258] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:15.887382 17473 tablet_replica.cc:330] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: stopping tablet replica
I20260927 15:23:15.887992 17473 raft_consensus.cc:2229] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:15.888631 17473 raft_consensus.cc:2258] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:15.888962 17471 ts_tablet_manager.cc:1905] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:15.891978 17473 ts_tablet_manager.cc:1905] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:15.892841 17472 ts_tablet_manager.cc:1905] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:15.899825 17471 ts_tablet_manager.cc:1918] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:15.900261 17471 log.cc:1196] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/48f6c0db4b5547ee8ab406f33114e261
Sep 27 15:23:15 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:23:15.901093 17471 ts_tablet_manager.cc:1939] T 48f6c0db4b5547ee8ab406f33114e261 P c8f2265220d7413fb3092f0b8364db10: Deleting consensus metadata
I20260927 15:23:15.904096 16957 catalog_manager.cc:4841] TS c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205): tablet 48f6c0db4b5547ee8ab406f33114e261 (table test-table [id=afcede8e8350420a8a2e4ffe164326f2]) successfully deleted
I20260927 15:23:15.907024 17473 ts_tablet_manager.cc:1918] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:15.907215 17472 ts_tablet_manager.cc:1918] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:15.907239 17473 log.cc:1196] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/48f6c0db4b5547ee8ab406f33114e261
I20260927 15:23:15.907430 17472 log.cc:1196] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/48f6c0db4b5547ee8ab406f33114e261
I20260927 15:23:15.907874 17473 ts_tablet_manager.cc:1939] T 48f6c0db4b5547ee8ab406f33114e261 P c4932dedb5234a3b9d6ee1cf295889cc: Deleting consensus metadata
I20260927 15:23:15.908085 17472 ts_tablet_manager.cc:1939] T 48f6c0db4b5547ee8ab406f33114e261 P cf4f3b83fcee46758423d474d41defaa: Deleting consensus metadata
I20260927 15:23:15.910087 16955 catalog_manager.cc:4841] TS c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409): tablet 48f6c0db4b5547ee8ab406f33114e261 (table test-table [id=afcede8e8350420a8a2e4ffe164326f2]) successfully deleted
I20260927 15:23:15.910385 16956 catalog_manager.cc:4841] TS cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241): tablet 48f6c0db4b5547ee8ab406f33114e261 (table test-table [id=afcede8e8350420a8a2e4ffe164326f2]) successfully deleted
I20260927 15:23:15.913106 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:15.878567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.2.246.65:36205)
0927 15:23:15.878915 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:15.878927 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:15.886658 (+ 7731us) server_negotiation.cc:407] Connection header received
0927 15:23:15.886929 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:15.886936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:15.887057 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:15.887232 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:15.890027 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.891386 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.894038 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:15.894882 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:15.895019 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:15.902218 (+ 7199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:15.902234 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:15.902257 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:15.902321 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:15.908965 (+ 6644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.910208 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.910218 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.910302 (+ 84us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.910553 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:15.911082 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:15.911088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:15.911092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:15.911569 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:15.911963 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:15.912292 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:23:15.912586 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 15:23:15.914058 17155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:16.030462 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17032
I20260927 15:23:16.054173 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17166
Sep 27 15:23:16 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:23:16.078500 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:16.055207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.2.246.66:40409)
0927 15:23:16.055397 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:16.055401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:16.055451 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:23:16.055753 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:16.055758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:16.055831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:16.055950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:16.057693 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:16.058635 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:16.060197 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:16.060604 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:16.060730 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:16.071882 (+ 11152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:16.071900 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:16.071919 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:16.071970 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:16.075167 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:16.076255 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:16.076263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:16.076265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:16.076414 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:16.076782 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:16.076785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:16.076787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:16.077209 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:16.077465 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:16.078040 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 15:23:16.078233 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:23:16.079540 17289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:16.184268 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17166
I20260927 15:23:16.203104 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17300
Sep 27 15:23:16 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:23:16.221076 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:16.204225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.2.246.67:42241)
0927 15:23:16.204453 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:16.204460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:16.204485 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:23:16.204672 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:16.204677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:16.204753 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:16.204887 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:16.206334 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:16.207152 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:16.208573 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:16.209029 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:16.209169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:16.214438 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:16.214460 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:16.214481 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:16.214532 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:16.217666 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:16.218690 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:16.218698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:16.218701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:16.218840 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:16.219254 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:16.219264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:16.219266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:16.219688 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:16.220049 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:16.220395 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:23:16.220648 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:23:16.222122 17423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:16.323813 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17300
I20260927 15:23:16.344537 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 16939
I20260927 15:23:16.346086 17001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:16.466333 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 16939
I20260927 15:23:16.495003 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--webserver_interface=127.2.246.126
--webserver_port=33687
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:16.894048 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:16.894403 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:16.894596 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:16.902211 17479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:16.902380 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:16.902494 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:16.902593 17479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:16.902683 17479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:16.913235 17479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=33687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:16.914860 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:16.916252 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:16.944833 17488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:16.944886 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:16.945153 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:16.946118 17479 server_base.cc:1034] running on GCE node
I20260927 15:23:16.947185 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:16.949067 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:16.950349 17479 hybrid_clock.cc:648] HybridClock initialized: now 1790522596950272 us; error 65 us; skew 500 ppm
I20260927 15:23:16.950947 17479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:16 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:16.955200 17479 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:16.957494 17479 webserver.cc:460] Webserver started at http://127.2.246.126:33687/ using document root <none> and password file <none>
I20260927 15:23:16.958465 17479 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:16.958655 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:16.966132 17479 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:23:16.970366 17495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:16.972170 17479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:16.972426 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "89b1dcd424084bf394170ecd3cd6a6c9"
format_stamp: "Formatted at 2026-09-27 15:23:13 on dist-test-slave-14jq"
server_key: "89df0c0c03e1fba87dca4276aff79573"
server_key_iv: "926f978bcd8551f92ea1c69a7cd32691"
server_key_version: "encryptionkey@0"
I20260927 15:23:16.973604 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:16.998275 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:17.003363 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:17.003718 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:17.026763 17479 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33997
I20260927 15:23:17.026865 17546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33997 every 8 connection(s)
I20260927 15:23:17.028491 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:17.038022 17547 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:23:17.038568 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17479
I20260927 15:23:17.039419 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:36205
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=46547
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:17.042935 17547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap starting.
I20260927 15:23:17.070734 17547 log.cc:824] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:17.087949 17547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:23:17.088722 17547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap complete.
I20260927 15:23:17.100575 17547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:17.101759 17547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Initialized, Role: FOLLOWER
I20260927 15:23:17.103029 17547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:17.103448 17547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:23:17.103734 17547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:23:17.104027 17547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:23:17.109308 17547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:17.110214 17547 leader_election.cc:302] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b1dcd424084bf394170ecd3cd6a6c9; no voters:
I20260927 15:23:17.111086 17547 leader_election.cc:288] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:23:17.111367 17551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:23:17.115456 17551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 LEADER]: Becoming Leader. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Running, Role: LEADER
I20260927 15:23:17.116328 17551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:17.116905 17547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:17.119530 17553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b1dcd424084bf394170ecd3cd6a6c9. Latest consensus state: current_term: 2 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:17.120535 17553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:17.120018 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:17.121235 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:17.131394 17557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:17.147068 17557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:17.149746 17557 catalog_manager.cc:1259] Loaded cluster ID: 6fa88ba46b1a442882bc2c63609bc0b9
I20260927 15:23:17.150106 17557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:23:17.155757 17557 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:23:17.175079 17557 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:23:17.196367 17557 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:23:17.199434 17557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Loaded TSK: 0
I20260927 15:23:17.201396 17557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:23:17.490878 17549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:17.491338 17549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:17.491518 17549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:17.500201 17549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:17.500442 17549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:17.500645 17549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:23:17.512822 17549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:36205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=46547
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:17.514768 17549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:17.516585 17549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:17.535386 17575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:17.535738 17576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:17.536651 17578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:17.536878 17549 server_base.cc:1034] running on GCE node
I20260927 15:23:17.537732 17549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:17.539165 17549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:17.540411 17549 hybrid_clock.cc:648] HybridClock initialized: now 1790522597540339 us; error 79 us; skew 500 ppm
I20260927 15:23:17.540814 17549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:17 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:17.544963 17549 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:23:17.547542 17549 webserver.cc:460] Webserver started at http://127.2.246.65:46547/ using document root <none> and password file <none>
I20260927 15:23:17.548454 17549 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:17.548640 17549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:17.556835 17549 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:23:17.560717 17585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:17.562873 17549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:23:17.563174 17549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c8f2265220d7413fb3092f0b8364db10"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "e3650af3ea02db239d2f2ce90a729ac7"
server_key_iv: "aa68449e16fd87667e42f3355bbecdac"
server_key_version: "encryptionkey@0"
I20260927 15:23:17.564565 17549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:17.583792 17549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:17.591073 17549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:17.591466 17549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:17.592840 17549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:17.594753 17549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:17.594918 17549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:17.595069 17549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:17.595170 17549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:17.644019 17549 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:36205
I20260927 15:23:17.644155 17697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:36205 every 8 connection(s)
I20260927 15:23:17.646122 17549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:23:17.648842 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17549
I20260927 15:23:17.649721 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:40409
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=34693
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:23:17 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:17.682600 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:17.650284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:45349)
0927 15:23:17.654561 (+ 4277us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:17.654624 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:17.656710 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:17.659138 (+ 2428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:17.659155 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:17.660091 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:17.661066 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:17.661099 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:17.663388 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:17.663398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:17.665910 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:17.665934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:17.667490 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:17.667499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:17.667886 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:17.668840 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:17.668878 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:17.672389 (+ 3511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:17.676684 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:17.676697 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:17.676725 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:17.677288 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:17.677928 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:17.677934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:17.677938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:17.678208 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:17.679623 (+ 1415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:17.679632 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:17.680096 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 15:23:17.680419 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 15:23:17.680976 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3914,"thread_start_us":3650,"threads_started":1}
I20260927 15:23:17.682813 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:17.650038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:45349 (local address 127.2.246.126:33997)
0927 15:23:17.650872 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:17.650894 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:17.654865 (+ 3971us) server_negotiation.cc:407] Connection header received
0927 15:23:17.657630 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:17.657661 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:17.658321 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:17.658776 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:17.661439 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:17.663115 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:17.666218 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:17.667163 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:17.667356 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:17.672780 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:17.672791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:17.672812 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:17.672868 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:17.676455 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:17.677484 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:17.677494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:17.677500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:17.677710 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:17.678445 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:17.678454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:17.678458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:17.679415 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:17.679676 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:17.680765 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0927 15:23:17.681337 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":335,"threads_started":1}
I20260927 15:23:17.686638 17698 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:17.687059 17698 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:17.688138 17698 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
I20260927 15:23:17.691251 17512 ts_manager.cc:194] Registered new tserver with Master: c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205)
I20260927 15:23:17.693044 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:45349
W20260927 15:23:18.013768 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:18.014173 17702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:18.014331 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:18.022398 17702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:18.022531 17702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:18.022717 17702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:23:18.034030 17702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:40409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=34693
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:18.035420 17702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:18.036763 17702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:18.051290 17708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:18.051363 17710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:18.051419 17707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:18.053130 17702 server_base.cc:1034] running on GCE node
I20260927 15:23:18.053826 17702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:18.054987 17702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:18.056265 17702 hybrid_clock.cc:648] HybridClock initialized: now 1790522598056174 us; error 81 us; skew 500 ppm
I20260927 15:23:18.056655 17702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:18 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:18.060575 17702 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:23:18.062500 17702 webserver.cc:460] Webserver started at http://127.2.246.66:34693/ using document root <none> and password file <none>
I20260927 15:23:18.063318 17702 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:18.063493 17702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:18.070009 17702 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 15:23:18.073593 17717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.075500 17702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:23:18.075811 17702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "f079366c01476c87216b53a7b6bb0fee"
server_key_iv: "8a2dbf1c99ce4e6c2f9a646141877b33"
server_key_version: "encryptionkey@0"
I20260927 15:23:18.076929 17702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:18.102648 17702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:18.107719 17702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:18.108052 17702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:18.109470 17702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:18.110970 17702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:18.111102 17702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.111238 17702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:18.111326 17702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.153716 17702 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:40409
I20260927 15:23:18.153822 17829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:40409 every 8 connection(s)
I20260927 15:23:18.155407 17702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:23:18.160823 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17702
I20260927 15:23:18.161463 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:42241
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=43071
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:23:18 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:18.183090 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.158828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:60707 (local address 127.2.246.126:33997)
0927 15:23:18.159160 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.159166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.160341 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 15:23:18.162752 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.162761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.162915 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.163071 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:18.165053 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.165950 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.167822 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.168321 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.168486 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.174313 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:18.174322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:18.174326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:18.174380 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:18.177764 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.179557 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.179566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.179569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.179792 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.180541 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.180547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.180552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.181006 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:18.181164 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.182453 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.182762 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 15:23:18.184785 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.159277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:60707)
0927 15:23:18.160127 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.160183 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.162405 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.163369 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.163379 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.164057 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:18.164808 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.164828 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.166144 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.166149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.167626 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.167644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.168654 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.168665 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.169160 (+ 495us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.170377 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:18.170418 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:18.173773 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:18.178349 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:18.178366 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:18.178391 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:18.179344 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:18.179965 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:18.179973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:18.179980 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:18.180315 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:18.181180 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:18.181193 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:18.181884 (+ 691us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.182311 (+ 427us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.183045 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":303,"threads_started":1}
I20260927 15:23:18.187198 17830 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:18.187652 17830 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:18.188620 17830 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
I20260927 15:23:18.190622 17512 ts_manager.cc:194] Registered new tserver with Master: c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409)
I20260927 15:23:18.191650 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:60707
W20260927 15:23:18.510335 17833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:18.510650 17833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:18.510812 17833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:18.518573 17833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:18.518711 17833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:18.518895 17833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:23:18.529089 17833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=43071
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:18.530581 17833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:18.531934 17833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:18.547669 17839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:18.547796 17838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:18.547950 17841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:18.548816 17833 server_base.cc:1034] running on GCE node
I20260927 15:23:18.549630 17833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:18.551321 17833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:18.552616 17833 hybrid_clock.cc:648] HybridClock initialized: now 1790522598552551 us; error 47 us; skew 500 ppm
I20260927 15:23:18.553119 17833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:18 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:18.557332 17833 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:23:18.559500 17833 webserver.cc:460] Webserver started at http://127.2.246.67:43071/ using document root <none> and password file <none>
I20260927 15:23:18.560446 17833 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:18.560586 17833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:18.567734 17833 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:23:18.571270 17848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.573333 17833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:18.573554 17833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "cf4f3b83fcee46758423d474d41defaa"
format_stamp: "Formatted at 2026-09-27 15:23:15 on dist-test-slave-14jq"
server_key: "c14dbcb32dcbb03035aba0f92bd1817f"
server_key_iv: "4b66ee6423dad09aba4bdd6929675664"
server_key_version: "encryptionkey@0"
I20260927 15:23:18.574699 17833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:18.609421 17833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:18.614502 17833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:18.614849 17833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:18.616034 17833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:18.617650 17833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:18.617743 17833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.617931 17833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:18.618034 17833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:18.665673 17833 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:42241
I20260927 15:23:18.665776 17960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:42241 every 8 connection(s)
I20260927 15:23:18.667326 17833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:23:18.669719 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 17833
Sep 27 15:23:18 dist-test-slave-14jq krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:18.687026 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.671132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.2.246.126:33997)
0927 15:23:18.672481 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.672487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.672507 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:23:18.672578 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.672583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.672706 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.672839 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:18.674554 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.675347 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.676728 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.677168 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.677386 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.680718 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:18.680726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:18.680732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:18.680780 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:18.684004 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.684738 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.684746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.684752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.684887 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.685262 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.685269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.685272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.685717 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:18.686017 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.686340 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.686575 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":1243,"thread_start_us":1169,"threads_started":1}
I20260927 15:23:18.695919 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.670965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:47609 (local address 127.2.246.126:33997)
0927 15:23:18.671164 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.671171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.673017 (+ 1846us) server_negotiation.cc:407] Connection header received
0927 15:23:18.675269 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.675276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.675385 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.675507 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:18.677257 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.678359 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.680874 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.681714 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.681898 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.686963 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:18.686971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:18.686975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:18.687024 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:18.691379 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.692732 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.692740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.692745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.692947 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:18.693665 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:18.693669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:18.693671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:18.694055 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:18.694212 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.695354 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.695576 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260927 15:23:18.697289 17698 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
I20260927 15:23:18.697801 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.671812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:47609)
0927 15:23:18.672788 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.672841 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.674964 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.675778 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.675788 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.676387 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:18.677071 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.677090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.678569 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.678574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.680678 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.680697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.682289 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.682295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.682570 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.683428 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:18.683457 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:18.686239 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:18.691636 (+ 5397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:18.691649 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:18.691670 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:18.692512 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:18.693146 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:18.693171 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:18.693177 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:18.693517 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:18.694232 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:18.694241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:18.694796 (+ 555us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.695221 (+ 425us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.696007 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":444,"threads_started":1}
I20260927 15:23:18.700081 17961 heartbeater.cc:347] Connected to a master server at 127.2.246.126:33997
I20260927 15:23:18.700537 17961 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:18.701519 17961 heartbeater.cc:511] Master 127.2.246.126:33997 requested a full tablet report, sending...
I20260927 15:23:18.703675 17512 ts_manager.cc:194] Registered new tserver with Master: cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241)
I20260927 15:23:18.704981 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:23:18.705000 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:47609
I20260927 15:23:18.713406 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.706314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.2.246.126:33997)
0927 15:23:18.706561 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.706568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.707145 (+ 577us) server_negotiation.cc:407] Connection header received
0927 15:23:18.707360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.707365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.707550 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.707823 (+ 273us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:23:18.708301 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.709118 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.711126 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.711522 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.711627 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.712221 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:23:18.712757 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:23:18.712898 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.713108 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.713198 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:23:18.714537 17512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:23:18.724013 17512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:23:18.769713 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.742785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:36205 (local address 127.0.0.1:53366)
0927 15:23:18.758877 (+ 16092us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.758900 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.759136 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.760166 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.760171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.760196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:18.760608 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.760619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.762551 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.762556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.766154 (+ 3598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.766171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.768141 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.768148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.768417 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.768461 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.768869 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.769157 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15907,"mutex_wait_us":814,"thread_start_us":9219,"threads_started":1}
I20260927 15:23:18.769652 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.743689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.2.246.65:36205)
0927 15:23:18.758023 (+ 14334us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.758032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.759017 (+ 985us) server_negotiation.cc:407] Connection header received
0927 15:23:18.759338 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.759347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.759591 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.759790 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:18.760891 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.762340 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.766356 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.767609 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.768038 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.768327 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.768940 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.769190 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14085,"thread_start_us":14321,"threads_started":1}
I20260927 15:23:18.770438 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.742042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:42241 (local address 127.0.0.1:46586)
0927 15:23:18.752622 (+ 10580us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.752692 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.752971 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.762057 (+ 9086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.762064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.762106 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:18.762739 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.762755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.764271 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.764278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.766529 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.766546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.769097 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.769104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.769289 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.769322 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.769641 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.769940 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10192,"mutex_wait_us":40,"thread_start_us":16424,"threads_started":1}
I20260927 15:23:18.770907 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.742729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.2.246.66:40409)
0927 15:23:18.757973 (+ 15244us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.757987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.758029 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:23:18.758183 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.758193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.758600 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.758872 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:18.760241 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.762006 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.765943 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.767653 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.767816 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.768088 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.769043 (+ 955us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.769265 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14963,"thread_start_us":16388,"threads_started":1}
I20260927 15:23:18.771134 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.742666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40409 (local address 127.0.0.1:45254)
0927 15:23:18.756983 (+ 14317us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.757017 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.757209 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.759398 (+ 2189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.759407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.759440 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:18.759973 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.759990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.762194 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.762201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.765645 (+ 3444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.765664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.768287 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.768294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.768482 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.768664 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.768798 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.770101 (+ 1303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14077,"mutex_wait_us":1236,"thread_start_us":13814,"threads_started":1}
I20260927 15:23:18.771155 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.742173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.2.246.67:42241)
0927 15:23:18.761355 (+ 19182us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.761366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.761403 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:23:18.761506 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.761513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.761733 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.762055 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:18.763020 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.764122 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.766758 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.768297 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.768449 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.768853 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.770505 (+ 1652us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.770741 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18964,"thread_start_us":18747,"threads_started":1}
I20260927 15:23:18.774497 17633 tablet_service.cc:1467] Processing CreateTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f (DEFAULT_TABLE table=test-table [id=15e4285187774bb5b1402b00c4a93ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:18.775894 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db46940b794b5f83c2cb05cbb9d54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:18.775804 17896 tablet_service.cc:1467] Processing CreateTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f (DEFAULT_TABLE table=test-table [id=15e4285187774bb5b1402b00c4a93ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:18.776475 17765 tablet_service.cc:1467] Processing CreateTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f (DEFAULT_TABLE table=test-table [id=15e4285187774bb5b1402b00c4a93ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:18.777622 17896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db46940b794b5f83c2cb05cbb9d54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:18.777766 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db46940b794b5f83c2cb05cbb9d54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:18.789109 17977 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Bootstrap starting.
I20260927 15:23:18.792474 17977 tablet_bootstrap.cc:654] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:18.793537 17977 log.cc:824] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:18.796361 17979 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Bootstrap starting.
I20260927 15:23:18.796700 17977 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: No bootstrap required, opened a new log
I20260927 15:23:18.797005 17977 ts_tablet_manager.cc:1397] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:23:18.798764 17980 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Bootstrap starting.
I20260927 15:23:18.800755 17979 tablet_bootstrap.cc:654] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:18.802320 17979 log.cc:824] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:18.803891 17980 tablet_bootstrap.cc:654] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:18.805503 17980 log.cc:824] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:18.806530 17979 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: No bootstrap required, opened a new log
I20260927 15:23:18.806871 17979 ts_tablet_manager.cc:1397] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 15:23:18.808317 17977 raft_consensus.cc:348] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.808808 17977 raft_consensus.cc:374] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:18.808908 17977 raft_consensus.cc:729] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f2265220d7413fb3092f0b8364db10, State: Initialized, Role: FOLLOWER
I20260927 15:23:18.809463 17980 tablet_bootstrap.cc:492] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: No bootstrap required, opened a new log
I20260927 15:23:18.809587 17977 consensus_queue.cc:260] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.809886 17980 ts_tablet_manager.cc:1397] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260927 15:23:18.811682 17977 ts_tablet_manager.cc:1428] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260927 15:23:18.816982 17979 raft_consensus.cc:348] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.817397 17979 raft_consensus.cc:374] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:18.817492 17979 raft_consensus.cc:729] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4f3b83fcee46758423d474d41defaa, State: Initialized, Role: FOLLOWER
I20260927 15:23:18.818262 17979 consensus_queue.cc:260] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.820101 17961 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
I20260927 15:23:18.820366 17979 ts_tablet_manager.cc:1428] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 15:23:18.821727 17980 raft_consensus.cc:348] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.822400 17980 raft_consensus.cc:374] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:18.822532 17980 raft_consensus.cc:729] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4932dedb5234a3b9d6ee1cf295889cc, State: Initialized, Role: FOLLOWER
I20260927 15:23:18.823423 17980 consensus_queue.cc:260] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.824879 17830 heartbeater.cc:503] Master 127.2.246.126:33997 was elected leader, sending a full tablet report...
I20260927 15:23:18.825958 17980 ts_tablet_manager.cc:1428] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
W20260927 15:23:18.899685 17699 tablet.cc:2369] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:18.907286 17984 raft_consensus.cc:482] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 15:23:18.907721 17831 tablet.cc:2369] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:18.907773 17984 raft_consensus.cc:504] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.910524 17984 leader_election.cc:288] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409), c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205)
I20260927 15:23:18.917711 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.911157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38597 (local address 127.2.246.65:36205)
0927 15:23:18.911443 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.911450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.911479 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:23:18.911646 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.911653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.911756 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.911910 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:18.912663 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.913504 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.915581 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.916492 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.916729 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.916834 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.917204 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.917372 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:23:18.917748 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.910957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:36205 (local address 127.2.246.67:38597)
0927 15:23:18.911281 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.911310 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.911463 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.912029 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.912036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.912075 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:18.912415 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.912429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.913720 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.913725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.915337 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.915357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.916771 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.916781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.917005 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.917035 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.917170 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.917352 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1}
I20260927 15:23:18.918841 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.910977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:40409 (local address 127.2.246.67:38901)
0927 15:23:18.912686 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:18.912704 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:18.912881 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:18.913223 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:18.913237 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:18.913265 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:18.913610 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.913619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.914754 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.914758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.916224 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:18.916238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.917598 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.917605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:18.917780 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.917954 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 15:23:18.918242 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 15:23:18.918403 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1576,"mutex_wait_us":536,"thread_start_us":734,"threads_started":1}
I20260927 15:23:18.918870 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.911168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38901 (local address 127.2.246.66:40409)
0927 15:23:18.911445 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.911452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.912797 (+ 1345us) server_negotiation.cc:407] Connection header received
0927 15:23:18.913022 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.913030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.913098 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.913196 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:18.913776 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.914583 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.916386 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.917374 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.917576 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.917656 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.918343 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.918551 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 15:23:18.919353 17962 tablet.cc:2369] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:18.919294 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db46940b794b5f83c2cb05cbb9d54f" candidate_uuid: "cf4f3b83fcee46758423d474d41defaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f2265220d7413fb3092f0b8364db10" is_pre_election: true
I20260927 15:23:18.920231 17653 raft_consensus.cc:2454] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4f3b83fcee46758423d474d41defaa in term 0.
I20260927 15:23:18.920581 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db46940b794b5f83c2cb05cbb9d54f" candidate_uuid: "cf4f3b83fcee46758423d474d41defaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" is_pre_election: true
I20260927 15:23:18.921200 17785 raft_consensus.cc:2454] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4f3b83fcee46758423d474d41defaa in term 0.
I20260927 15:23:18.921387 17851 leader_election.cc:302] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f2265220d7413fb3092f0b8364db10, cf4f3b83fcee46758423d474d41defaa; no voters:
I20260927 15:23:18.921958 17984 raft_consensus.cc:2781] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:23:18.922144 17984 raft_consensus.cc:482] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:23:18.922281 17984 raft_consensus.cc:3037] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:18.924628 17984 raft_consensus.cc:504] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.925444 17984 leader_election.cc:288] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [CANDIDATE]: Term 1 election: Requested vote from peers c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409), c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205)
I20260927 15:23:18.926183 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db46940b794b5f83c2cb05cbb9d54f" candidate_uuid: "cf4f3b83fcee46758423d474d41defaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
I20260927 15:23:18.926438 17785 raft_consensus.cc:3037] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:18.926564 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db46940b794b5f83c2cb05cbb9d54f" candidate_uuid: "cf4f3b83fcee46758423d474d41defaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f2265220d7413fb3092f0b8364db10"
I20260927 15:23:18.926997 17653 raft_consensus.cc:3037] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:18.928992 17785 raft_consensus.cc:2454] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4f3b83fcee46758423d474d41defaa in term 1.
I20260927 15:23:18.929561 17849 leader_election.cc:302] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4932dedb5234a3b9d6ee1cf295889cc, cf4f3b83fcee46758423d474d41defaa; no voters:
I20260927 15:23:18.929931 17984 raft_consensus.cc:2781] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:18.930147 17653 raft_consensus.cc:2454] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4f3b83fcee46758423d474d41defaa in term 1.
I20260927 15:23:18.930546 17984 raft_consensus.cc:686] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 1 LEADER]: Becoming Leader. State: Replica: cf4f3b83fcee46758423d474d41defaa, State: Running, Role: LEADER
I20260927 15:23:18.931161 17984 consensus_queue.cc:237] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } }
I20260927 15:23:18.936426 17512 catalog_manager.cc:5495] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa reported cstate change: term changed from 0 to 1, leader changed from <none> to cf4f3b83fcee46758423d474d41defaa (127.2.246.67). New cstate: current_term: 1 leader_uuid: "cf4f3b83fcee46758423d474d41defaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4f3b83fcee46758423d474d41defaa" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 } health_report { overall_health: UNKNOWN } } }
I20260927 15:23:18.961357 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:18.955937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.2.246.67:42241)
0927 15:23:18.956193 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:18.956200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:18.956237 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:23:18.956358 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:18.956361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:18.956439 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:18.956603 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:23:18.956971 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.957670 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.959144 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:18.959549 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:18.959680 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:18.960228 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:23:18.960762 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:23:18.960914 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:18.960995 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:23:18.961079 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:23:18.972327 17785 raft_consensus.cc:1261] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Refusing update from remote peer cf4f3b83fcee46758423d474d41defaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:18.972327 17653 raft_consensus.cc:1261] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Refusing update from remote peer cf4f3b83fcee46758423d474d41defaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:18.973579 17987 consensus_queue.cc:1035] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4932dedb5234a3b9d6ee1cf295889cc" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:18.974080 17984 consensus_queue.cc:1035] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f2265220d7413fb3092f0b8364db10" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:19.010180 17511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:19.010569 17511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:19.015584 17511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Sending DeleteTablet for 3 replicas of tablet b6db46940b794b5f83c2cb05cbb9d54f
I20260927 15:23:19.017305 17765 tablet_service.cc:1514] Processing DeleteTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:19 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:45254
I20260927 15:23:19.018061 17633 tablet_service.cc:1514] Processing DeleteTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:19 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:53366
I20260927 15:23:19.018239 17896 tablet_service.cc:1514] Processing DeleteTablet for tablet b6db46940b794b5f83c2cb05cbb9d54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:19 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:46586
I20260927 15:23:19.019613 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17549
I20260927 15:23:19.027177 18001 tablet_replica.cc:330] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: stopping tablet replica
I20260927 15:23:19.027684 18001 raft_consensus.cc:2229] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:23:19.028322 18001 raft_consensus.cc:2258] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:19.031622 18000 tablet_replica.cc:330] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: stopping tablet replica
I20260927 15:23:19.032097 18000 raft_consensus.cc:2229] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:19.032207 17999 tablet_replica.cc:330] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: stopping tablet replica
I20260927 15:23:19.032647 18000 raft_consensus.cc:2258] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:19.033104 17999 raft_consensus.cc:2229] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:19.033936 17999 raft_consensus.cc:2258] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:19.035452 18000 ts_tablet_manager.cc:1905] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:19.036837 18001 ts_tablet_manager.cc:1905] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:19.039119 17999 ts_tablet_manager.cc:1905] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:19.042886 18000 ts_tablet_manager.cc:1918] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:19.043082 18000 log.cc:1196] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/b6db46940b794b5f83c2cb05cbb9d54f
I20260927 15:23:19.043740 18000 ts_tablet_manager.cc:1939] T b6db46940b794b5f83c2cb05cbb9d54f P c8f2265220d7413fb3092f0b8364db10: Deleting consensus metadata
I20260927 15:23:19.046072 17498 catalog_manager.cc:4841] TS c8f2265220d7413fb3092f0b8364db10 (127.2.246.65:36205): tablet b6db46940b794b5f83c2cb05cbb9d54f (table test-table [id=15e4285187774bb5b1402b00c4a93ffc]) successfully deleted
I20260927 15:23:19.047144 18001 ts_tablet_manager.cc:1918] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:19.047528 18001 log.cc:1196] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/b6db46940b794b5f83c2cb05cbb9d54f
I20260927 15:23:19.048414 18001 ts_tablet_manager.cc:1939] T b6db46940b794b5f83c2cb05cbb9d54f P cf4f3b83fcee46758423d474d41defaa: Deleting consensus metadata
I20260927 15:23:19.050907 17497 catalog_manager.cc:4841] TS cf4f3b83fcee46758423d474d41defaa (127.2.246.67:42241): tablet b6db46940b794b5f83c2cb05cbb9d54f (table test-table [id=15e4285187774bb5b1402b00c4a93ffc]) successfully deleted
I20260927 15:23:19.053730 17999 ts_tablet_manager.cc:1918] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:19.054253 17999 log.cc:1196] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/b6db46940b794b5f83c2cb05cbb9d54f
I20260927 15:23:19.054986 17999 ts_tablet_manager.cc:1939] T b6db46940b794b5f83c2cb05cbb9d54f P c4932dedb5234a3b9d6ee1cf295889cc: Deleting consensus metadata
I20260927 15:23:19.056962 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:19.020908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.2.246.65:36205)
0927 15:23:19.021142 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:19.021150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:19.021254 (+ 104us) server_negotiation.cc:407] Connection header received
0927 15:23:19.021548 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:19.021557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:19.021701 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:19.021884 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:19.025135 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.026883 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.030299 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.031393 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.031546 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:19.046201 (+ 14655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:19.046228 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:19.046257 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:19.046333 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:19.051441 (+ 5108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.052664 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.052676 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.052680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.052873 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.053374 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.053382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.053388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.054260 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:19.054650 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:19.056232 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0927 15:23:19.056526 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:23:19.057425 17496 catalog_manager.cc:4841] TS c4932dedb5234a3b9d6ee1cf295889cc (127.2.246.66:40409): tablet b6db46940b794b5f83c2cb05cbb9d54f (table test-table [id=15e4285187774bb5b1402b00c4a93ffc]) successfully deleted
I20260927 15:23:19.058014 17693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:19.170511 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17549
I20260927 15:23:19.195776 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17702
I20260927 15:23:19.215709 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:19.197036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.2.246.66:40409)
0927 15:23:19.197277 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:19.197283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:19.197433 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:23:19.197639 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:19.197645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:19.197741 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:19.197912 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:19.199558 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.200655 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.202369 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.203073 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.203239 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:19.207292 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:19.207327 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:19.207359 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:19.207418 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:19.211798 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.212829 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.212840 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.212845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.213015 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.213486 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.213491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.213495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.214071 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:19.214406 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:19.215014 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:23:19.215307 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:23:19.216776 17825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:19.327186 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17702
I20260927 15:23:19.348757 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17833
I20260927 15:23:19.369011 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:19.350161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.2.246.67:42241)
0927 15:23:19.350452 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:19.350461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:19.350490 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:23:19.350576 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:19.350584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:19.350691 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:19.350861 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:19.352495 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.353695 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.355585 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:19.356297 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:19.356475 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:19.359938 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:19.359960 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:19.359986 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:19.360051 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:19.364751 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.366204 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.366214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.366218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.366401 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:19.366968 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:19.366974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:19.366977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:19.367551 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:19.367880 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:19.368316 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 15:23:19.368596 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:23:19.370283 17956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:19.502182 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17833
I20260927 15:23:19.524355 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 17479
I20260927 15:23:19.525928 17542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:19.677603 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 17479
I20260927 15:23:19.706609 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--webserver_interface=127.2.246.126
--webserver_port=33687
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:20.061555 18006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:20.062029 18006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:20.062243 18006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:20.070273 18006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:20.070405 18006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:20.070451 18006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:20.070554 18006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:20.070627 18006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:20.081166 18006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=33687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:20.082724 18006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:20.084210 18006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:20.098084 18011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:20.098237 18014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:20.098286 18012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:20.099514 18006 server_base.cc:1034] running on GCE node
I20260927 15:23:20.100431 18006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:20.102190 18006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:20.103538 18006 hybrid_clock.cc:648] HybridClock initialized: now 1790522600103456 us; error 67 us; skew 500 ppm
I20260927 15:23:20.104187 18006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:20 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:20.112011 18006 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:20.114002 18006 webserver.cc:460] Webserver started at http://127.2.246.126:33687/ using document root <none> and password file <none>
I20260927 15:23:20.114732 18006 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:20.114846 18006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:20.121050 18006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:23:20.128695 18021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:20.130825 18006 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260927 15:23:20.131050 18006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "89b1dcd424084bf394170ecd3cd6a6c9"
format_stamp: "Formatted at 2026-09-27 15:23:13 on dist-test-slave-14jq"
server_key: "89df0c0c03e1fba87dca4276aff79573"
server_key_iv: "926f978bcd8551f92ea1c69a7cd32691"
server_key_version: "encryptionkey@0"
I20260927 15:23:20.132128 18006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:20.162127 18006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:20.167092 18006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:20.167340 18006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:20.190481 18006 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:33997
I20260927 15:23:20.190567 18072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:33997 every 8 connection(s)
I20260927 15:23:20.192657 18006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:20.197660 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18006
I20260927 15:23:20.199041 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:36205
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=46547
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:20.206262 18073 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:23:20.213290 18073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap starting.
I20260927 15:23:20.240574 18073 log.cc:824] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:20.261618 18073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:23:20.262598 18073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: Bootstrap complete.
I20260927 15:23:20.273972 18073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:20.275373 18073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Initialized, Role: FOLLOWER
I20260927 15:23:20.276517 18073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:20.276918 18073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:23:20.277057 18073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:23:20.277293 18073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:23:20.283779 18073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:20.284777 18073 leader_election.cc:302] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b1dcd424084bf394170ecd3cd6a6c9; no voters:
I20260927 15:23:20.285629 18073 leader_election.cc:288] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:23:20.285801 18077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:23:20.288910 18073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:20.290197 18077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [term 3 LEADER]: Becoming Leader. State: Replica: 89b1dcd424084bf394170ecd3cd6a6c9, State: Running, Role: LEADER
I20260927 15:23:20.292475 18077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } }
I20260927 15:23:20.299248 18078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:20.299876 18078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:20.300485 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b1dcd424084bf394170ecd3cd6a6c9. Latest consensus state: current_term: 3 leader_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b1dcd424084bf394170ecd3cd6a6c9" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 33997 } } }
I20260927 15:23:20.300913 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:20.318320 18091 catalog_manager.cc:1259] Loaded cluster ID: 6fa88ba46b1a442882bc2c63609bc0b9
I20260927 15:23:20.318476 18091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: loading cluster ID for follower catalog manager: success
W20260927 15:23:20.320925 18091 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 89b1dcd424084bf394170ecd3cd6a6c9: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260927 15:23:20.321105 18091 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260927 15:23:20.321265 18093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:20.327584 18093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:20.329094 18093 catalog_manager.cc:1259] Loaded cluster ID: 6fa88ba46b1a442882bc2c63609bc0b9
I20260927 15:23:20.329216 18093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:23:20.331282 18093 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0557efe9ad google::LogMessage::Fail() at ??:0
@ 0x7f0557f0188a google::LogMessage::SendToLog() at ??:0
@ 0x7f0557efe4de google::LogMessage::Flush() at ??:0
@ 0x7f0557effa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f056d935512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f056d932976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f056d9ba610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f056d9ba439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f055ab4a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f055ab4d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f055ab4d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f055ab28eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f055de026db start_thread at ??:0
@ 0x7f0556d9861f clone at ??:0
W20260927 15:23:20.624533 18075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:20.624888 18075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:20.625051 18075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:20.633944 18075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:20.634068 18075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:20.634248 18075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:23:20.644825 18075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:36205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=46547
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:20.646412 18075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:20.647851 18075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:20.663555 18115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:20.663664 18117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:20.663704 18114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:20.664812 18075 server_base.cc:1034] running on GCE node
I20260927 15:23:20.665470 18075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:20.666666 18075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:20.667994 18075 hybrid_clock.cc:648] HybridClock initialized: now 1790522600667938 us; error 40 us; skew 500 ppm
I20260927 15:23:20.668453 18075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:20 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:20.672405 18075 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:23:20.674346 18075 webserver.cc:460] Webserver started at http://127.2.246.65:46547/ using document root <none> and password file <none>
I20260927 15:23:20.675117 18075 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:20.675275 18075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:20.681569 18075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:23:20.685315 18124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:20.687737 18075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:23:20.688072 18075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "c8f2265220d7413fb3092f0b8364db10"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "e3650af3ea02db239d2f2ce90a729ac7"
server_key_iv: "aa68449e16fd87667e42f3355bbecdac"
server_key_version: "encryptionkey@0"
I20260927 15:23:20.689482 18075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:20.712350 18075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:20.718931 18075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:20.719304 18075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:20.720662 18075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:20.722617 18075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:20.722730 18075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:20.722872 18075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:20.722961 18075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:20.765786 18075 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:36205
I20260927 15:23:20.765893 18236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:36205 every 8 connection(s)
I20260927 15:23:20.767649 18075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:23:20.773486 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:20.771120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:54225)
0927 15:23:20.772060 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:20.772366 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
W20260927 15:23:20.774338 18237 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33997 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:33997: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
I20260927 15:23:20.775429 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:20.774984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:45529)
0927 15:23:20.775185 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:20.775292 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:23:20.775746 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18075
I20260927 15:23:20.776615 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:40409
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=34693
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:20.776808 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:20.776330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:41115)
0927 15:23:20.776517 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:20.776617 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 15:23:20.777113 18237 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:23:21.134057 18240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:21.134406 18240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:21.134506 18240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:21.142297 18240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:21.142441 18240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:21.142561 18240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:23:21.152920 18240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:40409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=34693
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:21.154390 18240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:21.155725 18240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:21.171636 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:21.171703 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:21.171684 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:21.173244 18240 server_base.cc:1034] running on GCE node
I20260927 15:23:21.174196 18240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:21.175395 18240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:21.176630 18240 hybrid_clock.cc:648] HybridClock initialized: now 1790522601176544 us; error 88 us; skew 500 ppm
I20260927 15:23:21.177021 18240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:21 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:21.180858 18240 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:23:21.182900 18240 webserver.cc:460] Webserver started at http://127.2.246.66:34693/ using document root <none> and password file <none>
I20260927 15:23:21.183689 18240 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:21.183809 18240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:21.190477 18240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:23:21.194288 18255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.196532 18240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:21.196750 18240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "c4932dedb5234a3b9d6ee1cf295889cc"
format_stamp: "Formatted at 2026-09-27 15:23:14 on dist-test-slave-14jq"
server_key: "f079366c01476c87216b53a7b6bb0fee"
server_key_iv: "8a2dbf1c99ce4e6c2f9a646141877b33"
server_key_version: "encryptionkey@0"
I20260927 15:23:21.197891 18240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:21.215317 18240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:21.220779 18240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:21.221154 18240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:21.222512 18240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:21.224545 18240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:21.224721 18240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.224886 18240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:21.225003 18240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.267930 18240 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:40409
I20260927 15:23:21.268076 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:40409 every 8 connection(s)
I20260927 15:23:21.269886 18240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:23:21.276814 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.274079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:39983)
0927 15:23:21.275302 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.275640 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":483,"threads_started":1}
I20260927 15:23:21.277418 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18240
W20260927 15:23:21.277755 18368 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33997 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:33997: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
I20260927 15:23:21.278179 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:42241
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:33997
--webserver_port=43071
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:21.278964 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.278554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:36267)
0927 15:23:21.278735 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.278827 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:23:21.279968 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:50435)
0927 15:23:21.279750 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.279839 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:23:21.280247 18368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:23:21.626036 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:21.626381 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:21.626521 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:21.634231 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:21.634366 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:21.634542 18371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:23:21.644793 18371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=43071
--tserver_master_addrs=127.2.246.126:33997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:21.646308 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:21.647651 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:21.661777 18379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:21.661723 18376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:21.661863 18377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:21.663216 18371 server_base.cc:1034] running on GCE node
I20260927 15:23:21.664078 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:21.665395 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:21.666639 18371 hybrid_clock.cc:648] HybridClock initialized: now 1790522601666587 us; error 51 us; skew 500 ppm
I20260927 15:23:21.667107 18371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:21 dist-test-slave-14jq krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:21.671077 18371 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:23:21.673055 18371 webserver.cc:460] Webserver started at http://127.2.246.67:43071/ using document root <none> and password file <none>
I20260927 15:23:21.674026 18371 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:21.674189 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:21.681501 18371 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:23:21.686045 18386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.688316 18371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:23:21.688627 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "cf4f3b83fcee46758423d474d41defaa"
format_stamp: "Formatted at 2026-09-27 15:23:15 on dist-test-slave-14jq"
server_key: "c14dbcb32dcbb03035aba0f92bd1817f"
server_key_iv: "4b66ee6423dad09aba4bdd6929675664"
server_key_version: "encryptionkey@0"
I20260927 15:23:21.689996 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:21.707048 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:21.713162 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:21.713575 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:21.714974 18371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:21.716761 18371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:21.716908 18371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.717067 18371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:21.717163 18371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:21.760025 18371 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:42241
I20260927 15:23:21.760080 18498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:42241 every 8 connection(s)
I20260927 15:23:21.762069 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:23:21.765230 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18371
I20260927 15:23:21.769816 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.766368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:58301)
0927 15:23:21.768251 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.768585 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1508,"thread_start_us":1234,"threads_started":1}
W20260927 15:23:21.770779 18499 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:33997 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:33997: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
I20260927 15:23:21.772017 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.771530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:44767)
0927 15:23:21.771752 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.771858 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 15:23:21.773039 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.772664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:43823)
0927 15:23:21.772803 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.772886 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:23:21.773299 18499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:23:21.779016 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:21.777881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:49233)
0927 15:23:21.778592 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:21.778710 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":365,"threads_started":1}
I20260927 15:23:22.288692 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:22.280905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:51531)
0927 15:23:22.285446 (+ 4541us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:22.285674 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4324,"thread_start_us":4104,"threads_started":1}
I20260927 15:23:22.775102 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:22.774052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:56017)
0927 15:23:22.774660 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:22.774803 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":301,"threads_started":1}
I20260927 15:23:22.781018 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:22.780140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:40253)
0927 15:23:22.780705 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:22.780832 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
I20260927 15:23:23.291213 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:23.290243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:50135)
0927 15:23:23.290847 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:23.290965 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":286,"threads_started":1}
I20260927 15:23:23.777427 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:23.776401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:38425)
0927 15:23:23.777108 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:23.777229 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":394,"threads_started":1}
I20260927 15:23:23.783160 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:23.782256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:49159)
0927 15:23:23.782854 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:23.782960 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":335,"threads_started":1}
I20260927 15:23:24.293481 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:24.292523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:49713)
0927 15:23:24.293132 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:24.293240 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":323,"threads_started":1}
I20260927 15:23:24.779750 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:24.778730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:49221)
0927 15:23:24.779364 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:24.779493 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
I20260927 15:23:24.785774 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:24.784786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:37377)
0927 15:23:24.785434 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:24.785528 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260927 15:23:25.295969 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:25.294801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:34743)
0927 15:23:25.295476 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:25.295758 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":328,"threads_started":1}
I20260927 15:23:25.781977 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:25.780982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:37585)
0927 15:23:25.781577 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:25.781697 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260927 15:23:25.788247 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:25.787127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:51203)
0927 15:23:25.787859 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:25.788011 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":373,"threads_started":1}
I20260927 15:23:26.298548 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:26.297370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:57359)
0927 15:23:26.298215 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:26.298321 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":513,"threads_started":1}
I20260927 15:23:26.784178 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:26.783116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:51885)
0927 15:23:26.783816 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:26.783950 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260927 15:23:26.790477 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:26.789556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:46471)
0927 15:23:26.790183 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:26.790297 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":352,"threads_started":1}
I20260927 15:23:27.301070 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:27.300068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:33101)
0927 15:23:27.300667 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:27.300806 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260927 15:23:27.786552 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:27.785524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:51865)
0927 15:23:27.786220 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:27.786349 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":383,"threads_started":1}
I20260927 15:23:27.792759 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:27.791671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:36213)
0927 15:23:27.792375 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:27.792496 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":269,"threads_started":1}
I20260927 15:23:28.303411 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:28.302474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:36257)
0927 15:23:28.303103 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:28.303190 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":283,"threads_started":1}
I20260927 15:23:28.788721 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:28.787744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:54707)
0927 15:23:28.788391 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:28.788492 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":267,"threads_started":1}
I20260927 15:23:28.795261 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:28.794141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:34867)
0927 15:23:28.794852 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:28.794987 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":399,"threads_started":1}
I20260927 15:23:29.305662 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:29.304597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:56045)
0927 15:23:29.305270 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:29.305396 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
I20260927 15:23:29.790998 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:29.790024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:34519)
0927 15:23:29.790659 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:29.790782 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":322,"threads_started":1}
I20260927 15:23:29.797305 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:29.796334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:54041)
0927 15:23:29.796967 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:29.797078 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":287,"threads_started":1}
I20260927 15:23:30.308313 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:30.307147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:37457)
0927 15:23:30.308002 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:30.308112 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":381,"threads_started":1}
I20260927 15:23:30.793295 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:30.792315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:48289)
0927 15:23:30.792929 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:30.793056 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260927 15:23:30.799355 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:30.798419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:56967)
0927 15:23:30.799078 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:30.799176 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":431,"threads_started":1}
I20260927 15:23:31.310549 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:31.309587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:37277)
0927 15:23:31.310250 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:31.310360 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":388,"threads_started":1}
I20260927 15:23:31.795559 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:31.794585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:41837)
0927 15:23:31.795206 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:31.795344 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":359,"threads_started":1}
I20260927 15:23:31.801474 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:31.800488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:59139)
0927 15:23:31.801137 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:31.801276 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":307,"threads_started":1}
I20260927 15:23:32.312865 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:32.311875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:51285)
0927 15:23:32.312498 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:32.312635 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":295,"threads_started":1}
I20260927 15:23:32.798365 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:32.797266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:51145)
0927 15:23:32.798023 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:32.798146 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":325,"threads_started":1}
I20260927 15:23:32.803920 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:32.802625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:51477)
0927 15:23:32.803489 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:32.803658 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":480,"threads_started":1}
I20260927 15:23:33.315470 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:33.314468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:35857)
0927 15:23:33.315158 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:33.315275 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":375,"threads_started":1}
I20260927 15:23:33.800704 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:33.799598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:33877)
0927 15:23:33.800332 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:33.800478 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":298,"threads_started":1}
I20260927 15:23:33.806290 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:33.805222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:54563)
0927 15:23:33.805890 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:33.806055 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":281,"threads_started":1}
I20260927 15:23:34.318125 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:34.316887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:52183)
0927 15:23:34.317608 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:34.317751 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":353,"threads_started":1}
I20260927 15:23:34.803100 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:34.801964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:38919)
0927 15:23:34.802668 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:34.802818 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":314,"threads_started":1}
I20260927 15:23:34.808210 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:34.807352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:39829)
0927 15:23:34.807912 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:34.808012 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":287,"threads_started":1}
I20260927 15:23:35.320499 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:35.319462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:50281)
0927 15:23:35.320091 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:35.320233 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":309,"threads_started":1}
I20260927 15:23:35.805245 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:35.804303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:38353)
0927 15:23:35.804885 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:35.805009 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":271,"threads_started":1}
I20260927 15:23:35.810781 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:35.809643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:49897)
0927 15:23:35.810421 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:35.810545 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":486,"threads_started":1}
I20260927 15:23:36.323480 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:36.322340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.66:48459)
0927 15:23:36.323096 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:36.323242 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":399,"threads_started":1}
I20260927 15:23:36.778123 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18075
I20260927 15:23:36.806018 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:36.779568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.2.246.65:36205)
0927 15:23:36.780344 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:36.780360 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:36.780401 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:23:36.782106 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:36.782116 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:36.782558 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:36.782878 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:36.784982 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:36.786337 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:36.787768 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:36.788492 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:36.788715 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:36.797598 (+ 8883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:36.797608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:36.797680 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:36.797743 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:36.801512 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:36.802758 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:36.802767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:36.802775 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:36.803029 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:36.803528 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:36.803531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:36.803534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:36.804296 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:36.804560 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:36.805131 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 15:23:36.805538 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":388,"threads_started":1}
I20260927 15:23:36.807682 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:36.806599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.67:58703)
0927 15:23:36.807264 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:36.807414 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":305,"threads_started":1}
I20260927 15:23:36.808383 18232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:36.898782 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:36.896996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:33997 (local address 127.2.246.65:58341)
0927 15:23:36.898104 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:36.898359 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":705,"threads_started":1}
I20260927 15:23:36.899549 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18075
I20260927 15:23:36.920115 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18240
I20260927 15:23:36.941325 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:36.921377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.2.246.66:40409)
0927 15:23:36.922203 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:36.922223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:36.922270 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:23:36.924231 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:36.924244 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:36.924838 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:36.925243 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:36.927061 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:36.928492 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:36.930114 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:36.930939 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:36.931106 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:36.934284 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:36.934295 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:36.934316 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:36.934365 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:36.937758 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:36.938583 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:36.938592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:36.938600 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:36.938803 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:36.939152 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:36.939155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:36.939157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:36.939817 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:36.940100 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:36.940548 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 15:23:36.940977 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":369,"threads_started":1}
I20260927 15:23:36.943188 18363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:37.034757 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18240
I20260927 15:23:37.054397 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18371
I20260927 15:23:37.079931 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:37.055864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.2.246.67:42241)
0927 15:23:37.056632 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:37.056651 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:37.056706 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:23:37.058751 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:37.058764 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:37.059368 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:37.059774 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:37.061435 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:37.063318 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:37.065113 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:37.066228 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:37.066445 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:37.069763 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:37.069775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:37.069802 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:37.069995 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:37.074934 (+ 4939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:37.076134 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:37.076144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:37.076154 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:37.076439 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:37.076972 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:37.076976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:37.076978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:37.077766 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:37.078253 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:37.078844 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 15:23:37.079519 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":348,"threads_started":1}
I20260927 15:23:37.081796 18494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:37.173333 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18371
2026-09-27T15:23:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25710 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:23:37 dist-test-slave-14jq krb5kdc[18557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:23:37 dist-test-slave-14jq krb5kdc[18557](info): set up 2 sockets
Sep 27 15:23:37 dist-test-slave-14jq krb5kdc[18557](info): commencing operation
krb5kdc: starting...
W20260927 15:23:38.350796 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:23:38 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:23:38.642987 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:39339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:39.085338 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:39.086081 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:39.086323 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:39.095564 18570 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:23:39.095747 18570 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:23:39.095793 18570 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:23:39.095842 18570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:23:39.095944 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:39.096098 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:39.096189 18570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:23:39.096287 18570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:23:39.110726 18570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:39339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:39.112675 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:39.114598 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:39.133673 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:39.133888 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:39.133683 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:39.134296 18570 server_base.cc:1034] running on GCE node
I20260927 15:23:39.135703 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:39.137998 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:39.139381 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790522619139289 us; error 62 us; skew 500 ppm
I20260927 15:23:39.140076 18570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:39 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:39.144778 18570 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:23:39.146929 18570 webserver.cc:460] Webserver started at http://127.2.246.126:40493/ using document root <none> and password file <none>
I20260927 15:23:39.147794 18570 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:39.148022 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:39.148389 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:39.153076 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "9396e5648b40465f98ca4f5f943a26fb"
format_stamp: "Formatted at 2026-09-27 15:23:39 on dist-test-slave-14jq"
server_key: "c1fc4a432fc4a34715e55d050f31644b"
server_key_iv: "fa143b7dc6a72d9a9c540f2f0f54c6fe"
server_key_version: "encryptionkey@0"
I20260927 15:23:39.154562 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "9396e5648b40465f98ca4f5f943a26fb"
format_stamp: "Formatted at 2026-09-27 15:23:39 on dist-test-slave-14jq"
server_key: "c1fc4a432fc4a34715e55d050f31644b"
server_key_iv: "fa143b7dc6a72d9a9c540f2f0f54c6fe"
server_key_version: "encryptionkey@0"
I20260927 15:23:39.163303 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260927 15:23:39.169611 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:39.172183 18570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:23:39.172538 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "9396e5648b40465f98ca4f5f943a26fb"
format_stamp: "Formatted at 2026-09-27 15:23:39 on dist-test-slave-14jq"
server_key: "c1fc4a432fc4a34715e55d050f31644b"
server_key_iv: "fa143b7dc6a72d9a9c540f2f0f54c6fe"
server_key_version: "encryptionkey@0"
I20260927 15:23:39.172878 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:39.224745 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:39.231118 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:39.231537 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:39.257452 18570 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:39339
I20260927 15:23:39.257484 18636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:39339 every 8 connection(s)
I20260927 15:23:39.259112 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:23:39.264981 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:39.266819 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18570
I20260927 15:23:39.267168 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:23:39.267940 3033 external_mini_cluster.cc:1428] Setting key ebd6606905ee896d3fcf772f251b4e61
I20260927 15:23:39.281471 18637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap starting.
Sep 27 15:23:39 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:39.287322 18637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:39.289516 18637 log.cc:824] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:39.295136 18637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: No bootstrap required, opened a new log
I20260927 15:23:39.298586 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:39.270598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.2.246.126:39339)
0927 15:23:39.272111 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:39.272129 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:39.272200 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:23:39.274589 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:39.274633 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:39.275263 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:39.275754 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:39.277484 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:39.279260 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:39.280767 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:39.281552 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:39.281787 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:39.287757 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:39.287769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:39.287790 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:39.287866 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:39.293081 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:39.294095 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:39.294104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:39.294113 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:39.294408 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:39.294872 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:39.294882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:39.294884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:39.295722 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:39.296149 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:39.296656 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 15:23:39.297213 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":772,"threads_started":1}
I20260927 15:23:39.307593 18637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:23:39.308836 18637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:39.309432 18637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Initialized, Role: FOLLOWER
I20260927 15:23:39.310387 18637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:23:39.310763 18637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:23:39.311193 18637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:23:39.311448 18637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:39.314468 18637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:23:39.315294 18637 leader_election.cc:302] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9396e5648b40465f98ca4f5f943a26fb; no voters:
I20260927 15:23:39.316097 18637 leader_election.cc:288] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:23:39.317044 18642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:39.318130 18642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 LEADER]: Becoming Leader. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Running, Role: LEADER
I20260927 15:23:39.318872 18642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:23:39.319180 18637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:23:39.322686 18644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9396e5648b40465f98ca4f5f943a26fb. Latest consensus state: current_term: 1 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:23:39.322443 18643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:23:39.323475 18644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:39.323766 18643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:23:39.333559 18650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:23:39.340512 18650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:23:39.353245 18650 catalog_manager.cc:1347] Generated new cluster ID: 32aabec4e8594b428cf9f7b9dc4bf40a
I20260927 15:23:39.353466 18650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:23:39.388391 18650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:23:39.390442 18650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:23:39.399669 18650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Generated new TSK 0
I20260927 15:23:39.401068 18650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:23:39.488117 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:39.945461 18663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:39.946020 18663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:39.946240 18663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:39.955798 18663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:39.955984 18663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:39.956214 18663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:23:39.968163 18663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:39.970067 18663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:39.971863 18663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:39.991572 18669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:39.992156 18671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:39.992285 18668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:39.992998 18663 server_base.cc:1034] running on GCE node
I20260927 15:23:39.993991 18663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:39.995311 18663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:39.996572 18663 hybrid_clock.cc:648] HybridClock initialized: now 1790522619996509 us; error 47 us; skew 500 ppm
I20260927 15:23:39.996964 18663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:39 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:40.001112 18663 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:23:40.003226 18663 webserver.cc:460] Webserver started at http://127.2.246.65:36257/ using document root <none> and password file <none>
I20260927 15:23:40.004097 18663 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:40.004235 18663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:40.004652 18663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:40.008842 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "3471a8661d89199214e0732f2bd10933"
server_key_iv: "4ac6b33e1ea902fc8c3d60f4db389445"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.009824 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "3471a8661d89199214e0732f2bd10933"
server_key_iv: "4ac6b33e1ea902fc8c3d60f4db389445"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.016379 18663 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 15:23:40.021253 18678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.023597 18663 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260927 15:23:40.023896 18663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "3471a8661d89199214e0732f2bd10933"
server_key_iv: "4ac6b33e1ea902fc8c3d60f4db389445"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.024166 18663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:40.047690 18663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:40.052641 18663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:40.052992 18663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:40.054242 18663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:40.055768 18663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:40.055856 18663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.055945 18663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:40.055994 18663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.098151 18663 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45785
I20260927 15:23:40.098279 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45785 every 8 connection(s)
I20260927 15:23:40.099594 18663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:23:40.104821 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18663
I20260927 15:23:40.105163 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:23:40.106148 3033 external_mini_cluster.cc:1428] Setting key 1e5b824c37a333b83eca590501fb2319
Sep 27 15:23:40 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:40.130612 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:40.104097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:40145 (local address 127.2.246.126:39339)
0927 15:23:40.104782 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:40.104789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:40.106143 (+ 1354us) server_negotiation.cc:407] Connection header received
0927 15:23:40.109032 (+ 2889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:40.109036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:40.109120 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:40.109242 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:40.112283 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.113100 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.116192 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.116895 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.117069 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:40.123188 (+ 6119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:40.123195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:40.123198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:40.123242 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:40.126642 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:40.127645 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:40.127653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:40.127659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:40.127816 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:40.128388 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:40.128396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:40.128398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:40.128710 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:40.128902 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:40.129924 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0927 15:23:40.130200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":322,"threads_started":1}
I20260927 15:23:40.131675 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:40.104531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:40145)
0927 15:23:40.105817 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:40.105944 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:40.108594 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:40.109673 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:40.109690 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:40.110773 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:40.111948 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:40.111986 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.113345 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.113354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:40.115918 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:40.115943 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.117422 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.117439 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:40.117935 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:40.119035 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:40.119086 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:40.122822 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:40.126904 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:40.126915 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:40.126933 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:40.127477 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:40.127953 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:40.127958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:40.127961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:40.128241 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:40.128879 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:40.128888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:40.129388 (+ 500us) client_negotiation.cc:769] Sending connection context
0927 15:23:40.129767 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 15:23:40.130417 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":466,"threads_started":1}
I20260927 15:23:40.133637 18791 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:23:40.134298 18791 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:40.135854 18791 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
I20260927 15:23:40.139199 18602 ts_manager.cc:194] Registered new tserver with Master: fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785)
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:23:40.140985 18602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:40145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:23:40.178987 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:23:40.585253 18797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:40.585580 18797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:40.585688 18797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:40.593218 18797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:40.593360 18797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:40.593480 18797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:23:40.603796 18797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:40.605175 18797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:40.606606 18797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:40.623160 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:40.623248 18802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:40.623203 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:40.624328 18797 server_base.cc:1034] running on GCE node
I20260927 15:23:40.625197 18797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:40.626699 18797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:40.628000 18797 hybrid_clock.cc:648] HybridClock initialized: now 1790522620627903 us; error 77 us; skew 500 ppm
I20260927 15:23:40.628583 18797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:40 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:40.633279 18797 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:23:40.635282 18797 webserver.cc:460] Webserver started at http://127.2.246.66:39205/ using document root <none> and password file <none>
I20260927 15:23:40.636070 18797 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:40.636240 18797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:40.636579 18797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:40.639734 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "8a3c45bdf784593445eca8e4105ee52a"
server_key_iv: "5f81308a026dc8e50d55b4b954fff0a9"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.640626 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "8a3c45bdf784593445eca8e4105ee52a"
server_key_iv: "5f81308a026dc8e50d55b4b954fff0a9"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.647486 18797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:23:40.651994 18812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.654165 18797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:23:40.654414 18797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "8a3c45bdf784593445eca8e4105ee52a"
server_key_iv: "5f81308a026dc8e50d55b4b954fff0a9"
server_key_version: "encryptionkey@0"
I20260927 15:23:40.654649 18797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:40.692484 18797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:40.697422 18797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:40.697791 18797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:40.699016 18797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:40.700656 18797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:40.700788 18797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.700965 18797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:40.701045 18797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:40.743294 18797 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:44371
I20260927 15:23:40.743402 18924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:44371 every 8 connection(s)
I20260927 15:23:40.744722 18797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:23:40.745100 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18797
I20260927 15:23:40.745350 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:23:40.746117 3033 external_mini_cluster.cc:1428] Setting key a0166f97ddae731e6fc682ce3a74cf00
Sep 27 15:23:40 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:40.773646 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:40.748021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:43735 (local address 127.2.246.126:39339)
0927 15:23:40.748556 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:40.748561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:40.750314 (+ 1753us) server_negotiation.cc:407] Connection header received
0927 15:23:40.752663 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:40.752673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:40.752793 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:40.752956 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:40.755449 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.756659 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.760008 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.760500 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.760630 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:40.766963 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:40.766970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:40.766973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:40.767021 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:40.770034 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:40.770962 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:40.770972 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:40.770975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:40.771095 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:40.771756 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:40.771760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:40.771762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:40.772050 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:40.772224 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:40.773016 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 15:23:40.773258 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":321,"threads_started":1}
I20260927 15:23:40.774745 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:40.748933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:43735)
0927 15:23:40.750078 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:40.750152 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:40.752264 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:40.753404 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:40.753415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:40.754293 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:40.755169 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:40.755197 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.756931 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.756936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:40.759684 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:40.759716 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:40.760882 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:40.760896 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:40.761563 (+ 667us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:40.762760 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:40.762810 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:40.766559 (+ 3749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:40.770247 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:40.770259 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:40.770282 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:40.770812 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:40.771229 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:40.771234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:40.771238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:40.771442 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:40.772195 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:40.772203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:40.772609 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 15:23:40.772901 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 15:23:40.773498 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":468,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:23:40.776573 18925 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:23:40.777026 18925 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:40.778012 18925 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:23:40.780227 18602 ts_manager.cc:194] Registered new tserver with Master: 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371)
I20260927 15:23:40.781307 18602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:43735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:23:40.817263 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:23:41.145967 18791 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
W20260927 15:23:41.264369 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:23:41.264843 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:23:41.265029 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:23:41.275558 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:23:41.275791 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:23:41.276005 18931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:23:41.288822 18931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:23:41.290875 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:23:41.292714 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:23:41.311836 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:41.312582 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:23:41.312582 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:23:41.313542 18931 server_base.cc:1034] running on GCE node
I20260927 15:23:41.314388 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:23:41.315594 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:23:41.316867 18931 hybrid_clock.cc:648] HybridClock initialized: now 1790522621316768 us; error 83 us; skew 500 ppm
I20260927 15:23:41.317334 18931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:23:41 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:23:41.321128 18931 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:23:41.323094 18931 webserver.cc:460] Webserver started at http://127.2.246.67:41635/ using document root <none> and password file <none>
I20260927 15:23:41.323931 18931 fs_manager.cc:359] Metadata directory not provided
I20260927 15:23:41.324048 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:23:41.324415 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:23:41.327858 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "7eaa55af4cc94abc8db42239a2c2987f"
format_stamp: "Formatted at 2026-09-27 15:23:41 on dist-test-slave-14jq"
server_key: "6cf01f1c77c307f0f598a504186f91b8"
server_key_iv: "68932e6afa51724ab2af1c1811113a0f"
server_key_version: "encryptionkey@0"
I20260927 15:23:41.328799 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "7eaa55af4cc94abc8db42239a2c2987f"
format_stamp: "Formatted at 2026-09-27 15:23:41 on dist-test-slave-14jq"
server_key: "6cf01f1c77c307f0f598a504186f91b8"
server_key_iv: "68932e6afa51724ab2af1c1811113a0f"
server_key_version: "encryptionkey@0"
I20260927 15:23:41.334923 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:23:41.339094 18946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:41.340961 18931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:23:41.341228 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "7eaa55af4cc94abc8db42239a2c2987f"
format_stamp: "Formatted at 2026-09-27 15:23:41 on dist-test-slave-14jq"
server_key: "6cf01f1c77c307f0f598a504186f91b8"
server_key_iv: "68932e6afa51724ab2af1c1811113a0f"
server_key_version: "encryptionkey@0"
I20260927 15:23:41.341440 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:23:41.401358 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:23:41.407025 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:23:41.407414 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:23:41.408689 18931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:23:41.410653 18931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:23:41.410792 18931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:41.410998 18931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:23:41.411090 18931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:23:41.462049 18931 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:36545
I20260927 15:23:41.462184 19058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:36545 every 8 connection(s)
I20260927 15:23:41.463732 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:23:41.471305 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 18931
I20260927 15:23:41.471623 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:23:41.472330 3033 external_mini_cluster.cc:1428] Setting key 46da35365de92ddadfb28f2e3245bb92
Sep 27 15:23:41 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:23:41.491375 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.467552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:51483 (local address 127.2.246.126:39339)
0927 15:23:41.468198 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.468205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.469480 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 15:23:41.471884 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.471890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.472006 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.472144 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:41.473998 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.474875 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.476929 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.477375 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.477528 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.482117 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:41.482125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:41.482134 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:41.482184 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:41.486187 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.487788 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.487796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.487800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.488075 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.488795 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.488800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.488803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.489272 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:41.489501 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.490599 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.490916 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":323,"threads_started":1}
I20260927 15:23:41.492702 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.468296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:51483)
0927 15:23:41.469259 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.469319 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.471487 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.472447 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.472488 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.473113 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:23:41.473767 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.473786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.475065 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.475071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.476740 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.476754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.477634 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.477645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.478046 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.478890 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:23:41.478914 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:23:41.481691 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:23:41.486506 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:41.486517 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:41.486537 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:41.487419 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:41.488312 (+ 893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:23:41.488319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:23:41.488322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:23:41.488619 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:23:41.489501 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:23:41.489515 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:23:41.490135 (+ 620us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.490488 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.491227 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":357,"threads_started":1}
I20260927 15:23:41.494782 19059 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:23:41.495240 19059 heartbeater.cc:464] Registering TS with master...
I20260927 15:23:41.496209 19059 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
I20260927 15:23:41.498232 18602 ts_manager.cc:194] Registered new tserver with Master: 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
I20260927 15:23:41.499228 18602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:51483
I20260927 15:23:41.503525 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:23:41.525133 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.509628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.2.246.126:39339)
0927 15:23:41.509968 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.509977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.510410 (+ 433us) server_negotiation.cc:407] Connection header received
0927 15:23:41.510673 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.510678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.510779 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.510904 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:41.512469 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.513294 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.514757 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.515182 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.515325 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.518364 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:41.518370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:41.518374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:41.518413 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:41.521806 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.522714 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.522720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.522726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.522873 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.523325 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.523333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.523343 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.523742 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:41.524055 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.524470 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.524756 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:23:41.530939 18602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:23:41.539263 18602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:23:41.582250 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.556577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.2.246.66:44371)
0927 15:23:41.572382 (+ 15805us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.572395 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.572440 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:23:41.572584 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.572594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.572912 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.573129 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:41.574298 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.575701 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.579039 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.580847 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.581043 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.581309 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.581627 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.581828 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15626,"thread_start_us":15525,"threads_started":1}
I20260927 15:23:41.582337 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.556355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44371 (local address 127.0.0.1:36224)
0927 15:23:41.568210 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.568292 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.568638 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.573410 (+ 4772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.573419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.573455 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:41.574050 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.574070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.575874 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.575882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.578809 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.578829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.581089 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.581096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.581312 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.581383 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.581543 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.581808 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11394,"mutex_wait_us":40,"thread_start_us":16729,"threads_started":1}
I20260927 15:23:41.583755 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.557080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.2.246.67:36545)
0927 15:23:41.572593 (+ 15513us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.572606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.573101 (+ 495us) server_negotiation.cc:407] Connection header received
0927 15:23:41.573231 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.573239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.573609 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.573867 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:41.575113 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.576958 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.579930 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.581341 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.581483 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.581734 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.583074 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.583294 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12392,"thread_start_us":12843,"threads_started":1}
I20260927 15:23:41.583895 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.570401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:36182)
0927 15:23:41.573194 (+ 2793us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.573224 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.573404 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.574576 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.574586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.574614 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:41.575250 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.575266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.576896 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.576903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.580559 (+ 3656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.580576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.582814 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.582822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.583015 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.583048 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.583139 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.583295 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2609,"mutex_wait_us":802,"thread_start_us":3129,"threads_started":1}
I20260927 15:23:41.584512 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.556560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36545 (local address 127.0.0.1:45586)
0927 15:23:41.572749 (+ 16189us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.572775 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.572928 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.574206 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.574211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.574241 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:41.574827 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.574845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.577148 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.577155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.579720 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.579734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.581934 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.581942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.582142 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.582187 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.583988 (+ 1801us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.584157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16053,"mutex_wait_us":6,"thread_start_us":12368,"threads_started":1}
I20260927 15:23:41.586887 18860 tablet_service.cc:1467] Processing CreateTablet for tablet c8b718a0bd7549eca70a149d79a9af97 (DEFAULT_TABLE table=test-table [id=2685caa4b7b2492facdca5e8fcab56d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:41.588197 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.570969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.2.246.65:45785)
0927 15:23:41.573480 (+ 2511us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.573492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.573529 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:23:41.573667 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.573676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.574019 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.574257 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:41.575529 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.576703 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.580789 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.582282 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.582415 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.582648 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.587638 (+ 4990us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.587849 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":1206,"threads_started":1}
I20260927 15:23:41.588788 18860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b718a0bd7549eca70a149d79a9af97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:41.588526 18994 tablet_service.cc:1467] Processing CreateTablet for tablet c8b718a0bd7549eca70a149d79a9af97 (DEFAULT_TABLE table=test-table [id=2685caa4b7b2492facdca5e8fcab56d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:41.591007 18994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b718a0bd7549eca70a149d79a9af97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:41.592962 18726 tablet_service.cc:1467] Processing CreateTablet for tablet c8b718a0bd7549eca70a149d79a9af97 (DEFAULT_TABLE table=test-table [id=2685caa4b7b2492facdca5e8fcab56d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:23:41.594790 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b718a0bd7549eca70a149d79a9af97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:23:41.605602 19078 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Bootstrap starting.
I20260927 15:23:41.608435 19079 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Bootstrap starting.
I20260927 15:23:41.610972 19078 tablet_bootstrap.cc:654] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:41.612097 19079 tablet_bootstrap.cc:654] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:41.612408 19080 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Bootstrap starting.
I20260927 15:23:41.613646 19079 log.cc:824] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:41.614656 19078 log.cc:824] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:41.616441 19080 tablet_bootstrap.cc:654] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Neither blocks nor log segments found. Creating new log.
I20260927 15:23:41.617702 19080 log.cc:824] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Log is configured to *not* fsync() on all Append() calls
I20260927 15:23:41.618280 19079 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: No bootstrap required, opened a new log
I20260927 15:23:41.618520 19078 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: No bootstrap required, opened a new log
I20260927 15:23:41.618762 19079 ts_tablet_manager.cc:1397] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 15:23:41.618805 19078 ts_tablet_manager.cc:1397] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260927 15:23:41.620991 19080 tablet_bootstrap.cc:492] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: No bootstrap required, opened a new log
I20260927 15:23:41.621302 19080 ts_tablet_manager.cc:1397] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 15:23:41.629763 19080 raft_consensus.cc:348] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.630280 19080 raft_consensus.cc:374] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:41.630395 19080 raft_consensus.cc:729] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5da2fe0dd24d1d8829d4c9c0673d48, State: Initialized, Role: FOLLOWER
I20260927 15:23:41.631104 19080 consensus_queue.cc:260] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.632112 19079 raft_consensus.cc:348] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.632728 19079 raft_consensus.cc:374] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:41.632858 19079 raft_consensus.cc:729] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eaa55af4cc94abc8db42239a2c2987f, State: Initialized, Role: FOLLOWER
I20260927 15:23:41.633245 19080 ts_tablet_manager.cc:1428] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 15:23:41.633669 19079 consensus_queue.cc:260] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.632112 19078 raft_consensus.cc:348] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.635087 19078 raft_consensus.cc:374] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:23:41.635250 19078 raft_consensus.cc:729] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebfbbd8289f4d2d8826cd31951fb5b4, State: Initialized, Role: FOLLOWER
I20260927 15:23:41.636309 19078 consensus_queue.cc:260] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.642788 18925 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
I20260927 15:23:41.643914 19079 ts_tablet_manager.cc:1428] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Time spent starting tablet: real 0.025s user 0.016s sys 0.007s
I20260927 15:23:41.643918 19078 ts_tablet_manager.cc:1428] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Time spent starting tablet: real 0.025s user 0.014s sys 0.006s
I20260927 15:23:41.644443 19059 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
I20260927 15:23:41.662715 19084 raft_consensus.cc:482] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:23:41.663066 19084 raft_consensus.cc:504] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.665580 19084 leader_election.cc:288] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
I20260927 15:23:41.683871 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.667426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:54133 (local address 127.2.246.66:44371)
0927 15:23:41.668076 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.668085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.676326 (+ 8241us) server_negotiation.cc:407] Connection header received
0927 15:23:41.676622 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.676629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.676726 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.676802 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:41.677757 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.678847 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.681370 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.682440 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.682638 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.682732 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.683319 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.683494 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260927 15:23:41.684217 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.672118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44371 (local address 127.2.246.65:54133)
0927 15:23:41.676204 (+ 4086us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.676222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.676433 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.677031 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.677037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.677067 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:41.677572 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.677583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.679010 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.679020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.681144 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.681159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.682770 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.682780 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.683063 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.683127 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.683269 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.683511 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3793,"mutex_wait_us":123,"thread_start_us":2972,"threads_started":1}
I20260927 15:23:41.684983 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.666296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:35647 (local address 127.2.246.67:36545)
0927 15:23:41.666558 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.666566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.675768 (+ 9202us) server_negotiation.cc:407] Connection header received
0927 15:23:41.676166 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.676176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.676287 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.676364 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:23:41.677445 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.678724 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.682464 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.683540 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.683745 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.683817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.684523 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.684710 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:23:41.685704 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.670179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36545 (local address 127.2.246.65:35647)
0927 15:23:41.675555 (+ 5376us) negotiation.cc:107] Waiting for socket to connect
0927 15:23:41.675588 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:23:41.675949 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:23:41.676618 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:23:41.676622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:23:41.676647 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:23:41.677252 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.677265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.678904 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.678911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.682176 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:23:41.682211 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.684086 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.684097 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:23:41.684308 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.684351 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:23:41.685231 (+ 880us) client_negotiation.cc:240] Negotiation successful
0927 15:23:41.685391 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5182,"mutex_wait_us":28,"thread_start_us":5506,"threads_started":1}
I20260927 15:23:41.685729 18880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b718a0bd7549eca70a149d79a9af97" candidate_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" is_pre_election: true
I20260927 15:23:41.686393 18880 raft_consensus.cc:2454] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe5da2fe0dd24d1d8829d4c9c0673d48 in term 0.
I20260927 15:23:41.687116 19014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b718a0bd7549eca70a149d79a9af97" candidate_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eaa55af4cc94abc8db42239a2c2987f" is_pre_election: true
I20260927 15:23:41.687680 19014 raft_consensus.cc:2454] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe5da2fe0dd24d1d8829d4c9c0673d48 in term 0.
I20260927 15:23:41.688155 18679 leader_election.cc:302] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ebfbbd8289f4d2d8826cd31951fb5b4, fe5da2fe0dd24d1d8829d4c9c0673d48; no voters:
I20260927 15:23:41.688992 19084 raft_consensus.cc:2781] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:23:41.689231 19084 raft_consensus.cc:482] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:23:41.689396 19084 raft_consensus.cc:3037] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:41.692248 19084 raft_consensus.cc:504] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.693109 19084 leader_election.cc:288] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 1 election: Requested vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
I20260927 15:23:41.693789 19014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b718a0bd7549eca70a149d79a9af97" candidate_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eaa55af4cc94abc8db42239a2c2987f"
I20260927 15:23:41.693953 18880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b718a0bd7549eca70a149d79a9af97" candidate_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
I20260927 15:23:41.694068 19014 raft_consensus.cc:3037] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:41.694161 18880 raft_consensus.cc:3037] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:23:41.697474 18880 raft_consensus.cc:2454] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe5da2fe0dd24d1d8829d4c9c0673d48 in term 1.
I20260927 15:23:41.697474 19014 raft_consensus.cc:2454] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe5da2fe0dd24d1d8829d4c9c0673d48 in term 1.
I20260927 15:23:41.698025 18681 leader_election.cc:302] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eaa55af4cc94abc8db42239a2c2987f, fe5da2fe0dd24d1d8829d4c9c0673d48; no voters:
I20260927 15:23:41.698467 19084 raft_consensus.cc:2781] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:23:41.699098 19084 raft_consensus.cc:686] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 LEADER]: Becoming Leader. State: Replica: fe5da2fe0dd24d1d8829d4c9c0673d48, State: Running, Role: LEADER
I20260927 15:23:41.699700 19084 consensus_queue.cc:237] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } }
I20260927 15:23:41.705137 18601 catalog_manager.cc:5495] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65). New cstate: current_term: 1 leader_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } health_report { overall_health: HEALTHY } } }
W20260927 15:23:41.715831 19060 tablet.cc:2369] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:23:41.747548 18926 tablet.cc:2369] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:23:41.751458 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.741665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.2.246.65:45785)
0927 15:23:41.741982 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.741987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.742003 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:23:41.742096 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.742100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.742180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.742388 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:23:41.743223 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.744427 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.747660 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.748597 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.748782 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.749582 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:23:41.750286 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:23:41.750428 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.751033 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.751132 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:23:41.765483 18880 raft_consensus.cc:1261] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Refusing update from remote peer fe5da2fe0dd24d1d8829d4c9c0673d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:41.766304 19014 raft_consensus.cc:1261] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Refusing update from remote peer fe5da2fe0dd24d1d8829d4c9c0673d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:23:41.766957 19089 consensus_queue.cc:1035] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:41.768175 19084 consensus_queue.cc:1035] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:23:41.789413 19092 mvcc.cc:205] Tried to move back new op lower bound from 7333980658734641152 to 7333980658486861824. Current Snapshot: MvccSnapshot[applied={T|T < 7333980658734641152}]
I20260927 15:23:41.798718 19095 mvcc.cc:205] Tried to move back new op lower bound from 7333980658734641152 to 7333980658486861824. Current Snapshot: MvccSnapshot[applied={T|T < 7333980658734641152}]
I20260927 15:23:41.813422 18601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:41.813742 18601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:23:41.819113 18601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Sending DeleteTablet for 3 replicas of tablet c8b718a0bd7549eca70a149d79a9af97
I20260927 15:23:41.820714 18860 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b718a0bd7549eca70a149d79a9af97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:41 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260927 15:23:41.820995 18994 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b718a0bd7549eca70a149d79a9af97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:41 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260927 15:23:41.822656 18993 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b718a0bd7549eca70a149d79a9af97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:41 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:45586
I20260927 15:23:41.823841 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b718a0bd7549eca70a149d79a9af97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:23:41 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:36182
I20260927 15:23:41.830368 19102 tablet_replica.cc:330] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: stopping tablet replica
I20260927 15:23:41.830994 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18663
I20260927 15:23:41.831141 19102 raft_consensus.cc:2229] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:41.831921 19102 raft_consensus.cc:2258] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:41.834784 19104 tablet_replica.cc:330] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: stopping tablet replica
I20260927 15:23:41.835451 19104 raft_consensus.cc:2229] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:23:41.836416 19104 raft_consensus.cc:2258] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:41.837924 19103 tablet_replica.cc:330] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: stopping tablet replica
I20260927 15:23:41.838671 19103 raft_consensus.cc:2229] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:23:41.839418 19103 raft_consensus.cc:2258] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:23:41.840101 19102 ts_tablet_manager.cc:1905] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:41.842262 19103 ts_tablet_manager.cc:1905] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:41.848896 19102 ts_tablet_manager.cc:1918] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:41.849176 19102 log.cc:1196] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/c8b718a0bd7549eca70a149d79a9af97
I20260927 15:23:41.849817 19103 ts_tablet_manager.cc:1918] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:41.849932 19102 ts_tablet_manager.cc:1939] T c8b718a0bd7549eca70a149d79a9af97 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting consensus metadata
I20260927 15:23:41.850219 19103 log.cc:1196] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/c8b718a0bd7549eca70a149d79a9af97
I20260927 15:23:41.850279 19104 ts_tablet_manager.cc:1905] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:23:41.850910 19103 ts_tablet_manager.cc:1939] T c8b718a0bd7549eca70a149d79a9af97 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting consensus metadata
I20260927 15:23:41.852521 18586 catalog_manager.cc:4841] TS 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): tablet c8b718a0bd7549eca70a149d79a9af97 (table test-table [id=2685caa4b7b2492facdca5e8fcab56d3]) successfully deleted
I20260927 15:23:41.853300 18588 catalog_manager.cc:4841] TS 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): tablet c8b718a0bd7549eca70a149d79a9af97 (table test-table [id=2685caa4b7b2492facdca5e8fcab56d3]) successfully deleted
W20260927 15:23:41.854820 18588 catalog_manager.cc:4808] TS 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): delete failed for tablet c8b718a0bd7549eca70a149d79a9af97 because the tablet was not found. No further retry: Not found: Tablet not found: c8b718a0bd7549eca70a149d79a9af97
Sep 27 15:23:41 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:23:41.867378 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:41.832325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.2.246.65:45785)
0927 15:23:41.832522 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:41.832528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:41.837946 (+ 5418us) server_negotiation.cc:407] Connection header received
0927 15:23:41.838342 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:41.838348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:41.838445 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:41.838540 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:41.841162 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.842437 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.844672 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:41.845436 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:41.845556 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:41.856891 (+ 11335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:41.856914 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:41.856935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:41.857005 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:41.862140 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.863330 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.863339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.863343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.863564 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:41.864135 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:41.864141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:41.864144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:41.864711 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:41.865074 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:41.866621 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0927 15:23:41.866934 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:23:41.868911 18786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:41.982328 19104 ts_tablet_manager.cc:1918] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:23:41.982776 19104 log.cc:1196] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/c8b718a0bd7549eca70a149d79a9af97
I20260927 15:23:41.983649 19104 ts_tablet_manager.cc:1939] T c8b718a0bd7549eca70a149d79a9af97 P fe5da2fe0dd24d1d8829d4c9c0673d48: Deleting consensus metadata
I20260927 15:23:41.986413 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18663
I20260927 15:23:41.987243 18588 catalog_manager.cc:4841] TS fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785): tablet c8b718a0bd7549eca70a149d79a9af97 (table test-table [id=2685caa4b7b2492facdca5e8fcab56d3]) successfully deleted
I20260927 15:23:42.008872 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18797
Sep 27 15:23:42 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:23:42.028450 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:42.009986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.2.246.66:44371)
0927 15:23:42.010182 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:42.010186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:42.010262 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:23:42.010450 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:42.010454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:42.010529 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:42.010701 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:42.012401 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:42.013253 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:42.015177 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:42.015603 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:42.015747 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:42.021784 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:42.021804 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:42.021821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:42.021939 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:42.025182 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:42.026204 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:42.026212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:42.026214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:42.026354 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:42.026764 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:42.026767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:42.026769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:42.027154 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:42.027473 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:42.027803 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:23:42.028042 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:23:42.029588 18920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:42.135210 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18797
I20260927 15:23:42.157142 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18931
Sep 27 15:23:42 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:23:42.176015 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:23:42.158273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.2.246.67:36545)
0927 15:23:42.158523 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:23:42.158529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:23:42.158552 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:23:42.158720 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:23:42.158725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:23:42.158801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:23:42.158932 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:23:42.160421 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:42.161216 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:42.162916 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:23:42.163320 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:23:42.163436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:23:42.168868 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:23:42.168889 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:23:42.168908 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:23:42.168956 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:23:42.172135 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:42.173089 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:42.173096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:42.173098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:42.173253 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:23:42.174041 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:23:42.174051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:23:42.174057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:23:42.174595 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:23:42.174914 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 15:23:42.175315 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 15:23:42.175560 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 15:23:42.176816 19054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:42.275177 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18931
I20260927 15:23:42.298437 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 18570
I20260927 15:23:42.299970 18632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:23:42.423367 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 18570
I20260927 15:24:02.450759 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--webserver_interface=127.2.246.126
--webserver_port=40493
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:39339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:02.809672 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:02.810056 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:02.810225 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:02.817698 19110 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:24:02.817879 19110 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:24:02.817977 19110 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:24:02.818078 19110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:02.818182 19110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:02.818286 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:02.818377 19110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:02.818456 19110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:02.828572 19110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:39339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=40493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:02.830060 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:02.831460 19110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:02.860282 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:02.860445 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:02.860406 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:02.861465 19110 server_base.cc:1034] running on GCE node
I20260927 15:24:02.862666 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:02.864645 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:02.866027 19110 hybrid_clock.cc:648] HybridClock initialized: now 1790522642865928 us; error 58 us; skew 500 ppm
I20260927 15:24:02.866528 19110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:02 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522642, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:02.878795 19110 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:24:02.881173 19110 webserver.cc:460] Webserver started at http://127.2.246.126:40493/ using document root <none> and password file <none>
I20260927 15:24:02.882143 19110 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:02.882315 19110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:02.889032 19110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 15:24:02.892699 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:02.894532 19110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:24:02.894747 19110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "9396e5648b40465f98ca4f5f943a26fb"
format_stamp: "Formatted at 2026-09-27 15:23:39 on dist-test-slave-14jq"
server_key: "c1fc4a432fc4a34715e55d050f31644b"
server_key_iv: "fa143b7dc6a72d9a9c540f2f0f54c6fe"
server_key_version: "encryptionkey@0"
I20260927 15:24:02.895838 19110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:02.912704 19110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:02.918427 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:02.918766 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:02.941828 19110 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:39339
I20260927 15:24:02.941998 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:39339 every 8 connection(s)
I20260927 15:24:02.943565 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:02.953631 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19110
I20260927 15:24:02.954053 19178 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:24:02.954900 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:45785
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=36257
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:24:02.960736 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap starting.
I20260927 15:24:02.989037 19178 log.cc:824] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:03.004905 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:24:03.005707 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap complete.
I20260927 15:24:03.017040 19178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:03.018218 19178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Initialized, Role: FOLLOWER
I20260927 15:24:03.019151 19178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:03.019438 19178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:03.019579 19178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:03.019794 19178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:24:03.026163 19178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:03.026950 19178 leader_election.cc:302] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9396e5648b40465f98ca4f5f943a26fb; no voters:
I20260927 15:24:03.027768 19178 leader_election.cc:288] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:24:03.028069 19182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:24:03.030189 19182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 LEADER]: Becoming Leader. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Running, Role: LEADER
I20260927 15:24:03.030975 19182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:03.031478 19178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:03.034353 19184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9396e5648b40465f98ca4f5f943a26fb. Latest consensus state: current_term: 2 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:24:03.034924 19184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:03.035601 19183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:24:03.035957 19183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:03.041896 19188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:03.055451 19188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:03.058553 19188 catalog_manager.cc:1259] Loaded cluster ID: 32aabec4e8594b428cf9f7b9dc4bf40a
I20260927 15:24:03.058858 19188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:03.065055 19188 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:03.068634 19188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Loaded TSK: 0
I20260927 15:24:03.070211 19199 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:24:03.098043 19188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Generated new TSK 1
I20260927 15:24:03.099086 19188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:24:03.419760 19180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:03.420212 19180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:03.420392 19180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:03.430161 19180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:03.430366 19180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:03.430585 19180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:03.443125 19180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:45785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=36257
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:03.445045 19180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:03.446884 19180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:03.466158 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:03.466145 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:03.466675 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:03.467715 19180 server_base.cc:1034] running on GCE node
I20260927 15:24:03.468448 19180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:03.469592 19180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:03.471082 19180 hybrid_clock.cc:648] HybridClock initialized: now 1790522643471001 us; error 76 us; skew 500 ppm
I20260927 15:24:03.471478 19180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:03 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:03.475286 19180 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:24:03.477471 19180 webserver.cc:460] Webserver started at http://127.2.246.65:36257/ using document root <none> and password file <none>
I20260927 15:24:03.478408 19180 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:03.478600 19180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:03.484994 19180 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:24:03.488752 19215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:03.490605 19180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:24:03.490820 19180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "3471a8661d89199214e0732f2bd10933"
server_key_iv: "4ac6b33e1ea902fc8c3d60f4db389445"
server_key_version: "encryptionkey@0"
I20260927 15:24:03.491935 19180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:03.528213 19180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:03.533231 19180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:03.533540 19180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:03.534705 19180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:03.536250 19180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:03.536391 19180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:03.536537 19180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:03.536629 19180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:03.580619 19180 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45785
I20260927 15:24:03.580981 19327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45785 every 8 connection(s)
I20260927 15:24:03.582644 19180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:03.592159 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19180
I20260927 15:24:03.593379 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:44371
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=39205
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:24:03 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:03.612210 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:03.587488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:40923)
0927 15:24:03.588473 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:03.588536 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:03.590570 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:03.592284 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:03.592294 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:03.592969 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:03.593551 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:03.593606 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:03.595557 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:03.595565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:03.597123 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:03.597143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:03.598495 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:03.598503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:03.598800 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:03.599638 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:03.599669 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:03.602922 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:03.607182 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:03.607193 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:03.607211 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:03.607750 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:03.608299 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:03.608304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:03.608308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:03.608525 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:03.609393 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:03.609402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:03.609777 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 15:24:03.610145 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 15:24:03.610795 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":379,"threads_started":1}
I20260927 15:24:03.612181 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:03.587136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:40923 (local address 127.2.246.126:39339)
0927 15:24:03.587795 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:03.587810 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:03.588705 (+ 895us) server_negotiation.cc:407] Connection header received
0927 15:24:03.591280 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:03.591293 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:03.591701 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:03.592105 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:03.593886 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:03.595297 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:03.597325 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:03.598177 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:03.598427 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:03.603315 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:03.603325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:03.603342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:03.603390 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:03.606973 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:03.607928 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:03.607939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:03.607947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:03.608150 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:03.608653 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:03.608660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:03.608666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:03.609220 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:03.609448 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:03.610371 (+ 923us) server_negotiation.cc:299] Negotiation successful
0927 15:24:03.610791 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":311,"threads_started":1}
I20260927 15:24:03.616154 19328 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:24:03.616583 19328 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:03.617528 19328 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
I20260927 15:24:03.620322 19143 ts_manager.cc:194] Registered new tserver with Master: fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785)
I20260927 15:24:03.621951 19143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:40923
W20260927 15:24:04.015381 19332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:04.015821 19332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:04.015980 19332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:04.023725 19332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:04.023901 19332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:04.024096 19332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:04.034597 19332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:44371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=39205
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:04.036074 19332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:04.037518 19332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:04.052989 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:04.053263 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:04.053514 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:04.054345 19332 server_base.cc:1034] running on GCE node
I20260927 15:24:04.055140 19332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:04.056360 19332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:04.057600 19332 hybrid_clock.cc:648] HybridClock initialized: now 1790522644057514 us; error 81 us; skew 500 ppm
I20260927 15:24:04.058048 19332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:04 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:04.062307 19332 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:24:04.064447 19332 webserver.cc:460] Webserver started at http://127.2.246.66:39205/ using document root <none> and password file <none>
I20260927 15:24:04.065416 19332 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:04.065594 19332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:04.072983 19332 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:24:04.077252 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:04.079578 19332 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:24:04.079878 19332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "8a3c45bdf784593445eca8e4105ee52a"
server_key_iv: "5f81308a026dc8e50d55b4b954fff0a9"
server_key_version: "encryptionkey@0"
I20260927 15:24:04.081079 19332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:04.097762 19332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:04.102890 19332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:04.103142 19332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:04.104210 19332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:04.105883 19332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:04.105988 19332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:24:04.106078 19332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:04.106142 19332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:04.148401 19332 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:44371
I20260927 15:24:04.148929 19459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:44371 every 8 connection(s)
I20260927 15:24:04.150193 19332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:04.157227 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19332
I20260927 15:24:04.158207 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:36545
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=41635
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:24:04 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:04.182878 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.153926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:42663 (local address 127.2.246.126:39339)
0927 15:24:04.154578 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.154584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.155686 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 15:24:04.159017 (+ 3331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.159025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.159138 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.159295 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:04.162659 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.163494 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.166826 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.167352 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.167523 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.172953 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:04.172963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:04.172972 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:04.173035 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:04.177631 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.179069 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.179078 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.179085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.179331 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.180034 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.180042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.180049 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.180495 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:04.180680 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.181972 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.182312 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":335,"threads_started":1}
I20260927 15:24:04.184355 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.154401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:42663)
0927 15:24:04.155420 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.155484 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.158418 (+ 2934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.159759 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.159780 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.160885 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:04.162232 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.162271 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.163787 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.163797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.166501 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.166534 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.167732 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.167744 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.168242 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.169290 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:04.169322 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:04.172521 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:04.178038 (+ 5517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:04.178051 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:04.178073 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:04.178820 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:04.179528 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:04.179532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:04.179535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:04.179846 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:04.180758 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:04.180768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:04.181383 (+ 615us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.181767 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.182681 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":394,"threads_started":1}
I20260927 15:24:04.186857 19460 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:24:04.187366 19460 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:04.188460 19460 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
I20260927 15:24:04.190897 19143 ts_manager.cc:194] Registered new tserver with Master: 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371)
I20260927 15:24:04.192059 19143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:42663
W20260927 15:24:04.529421 19464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:04.529739 19464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:04.529903 19464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:04.537669 19464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:04.537810 19464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:04.538074 19464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:04.548725 19464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=41635
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:04.550091 19464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:04.551357 19464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:04.567323 19469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:04.567308 19470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:04.567358 19472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:04.568183 19464 server_base.cc:1034] running on GCE node
I20260927 15:24:04.569206 19464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:04.570641 19464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:04.571959 19464 hybrid_clock.cc:648] HybridClock initialized: now 1790522644571898 us; error 47 us; skew 500 ppm
I20260927 15:24:04.572456 19464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:04 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:04.576756 19464 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:24:04.578768 19464 webserver.cc:460] Webserver started at http://127.2.246.67:41635/ using document root <none> and password file <none>
I20260927 15:24:04.579723 19464 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:04.579885 19464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:04.587488 19464 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:24:04.590992 19479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:04.592759 19464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:24:04.592940 19464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "7eaa55af4cc94abc8db42239a2c2987f"
format_stamp: "Formatted at 2026-09-27 15:23:41 on dist-test-slave-14jq"
server_key: "6cf01f1c77c307f0f598a504186f91b8"
server_key_iv: "68932e6afa51724ab2af1c1811113a0f"
server_key_version: "encryptionkey@0"
I20260927 15:24:04.594070 19464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:04.621726 19464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:04.625983 19328 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
I20260927 15:24:04.627039 19464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:04.627378 19464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:04.628676 19464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:04.630323 19464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:04.630451 19464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:04.630594 19464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:04.630697 19464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:04.671072 19464 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:36545
I20260927 15:24:04.671132 19591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:36545 every 8 connection(s)
I20260927 15:24:04.673028 19464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:04.674861 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19464
Sep 27 15:24:04 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:04.692417 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.676177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.2.246.126:39339)
0927 15:24:04.676413 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.676420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.676977 (+ 557us) server_negotiation.cc:407] Connection header received
0927 15:24:04.677238 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.677244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.677354 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.677475 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:04.679367 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.680179 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.681866 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.682301 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.682449 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.685532 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:04.685538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:04.685541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:04.685575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:04.689200 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.690064 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.690075 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.690078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.690214 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.690674 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.690679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.690682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.691089 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:04.691410 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.691713 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.692006 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:24:04.700425 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.676759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:57259 (local address 127.2.246.126:39339)
0927 15:24:04.677891 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.677899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.678826 (+ 927us) server_negotiation.cc:407] Connection header received
0927 15:24:04.681102 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.681108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.681196 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.681330 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:04.683411 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.684402 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.686826 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.687609 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.687779 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.692454 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:04.692459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:04.692463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:04.692527 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:04.696280 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.697297 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.697301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.697304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.697434 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.697998 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.698002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.698004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.698386 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:04.698534 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.699713 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.700030 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":399,"threads_started":1}
I20260927 15:24:04.701606 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.677527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:57259)
0927 15:24:04.678607 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.678665 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.680783 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.681609 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.681619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.682388 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:04.683142 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.683166 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.684577 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.684585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.686498 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.686518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.687894 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.687906 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.688212 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.689005 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:04.689036 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:04.692133 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:04.696435 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:04.696444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:04.696510 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:04.697118 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:04.697551 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:04.697555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:04.697558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:04.697816 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:04.698568 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:04.698576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:04.699175 (+ 599us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.699589 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.700192 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":468,"threads_started":1}
I20260927 15:24:04.703640 19592 heartbeater.cc:347] Connected to a master server at 127.2.246.126:39339
I20260927 15:24:04.704114 19592 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:04.704979 19592 heartbeater.cc:511] Master 127.2.246.126:39339 requested a full tablet report, sending...
I20260927 15:24:04.707439 19143 ts_manager.cc:194] Registered new tserver with Master: 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
I20260927 15:24:04.708860 19143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:57259
I20260927 15:24:04.709730 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:24:04.717900 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.710964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.2.246.126:39339)
0927 15:24:04.711232 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.711240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.711743 (+ 503us) server_negotiation.cc:407] Connection header received
0927 15:24:04.712082 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.712087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.712270 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.712445 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:04.712981 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.713785 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.715463 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.715921 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.716099 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.716732 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:04.717014 (+ 282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:24:04.717508 (+ 494us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40994: token expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:24:04.718091 19595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40994: token expired
W20260927 15:24:04.718647 19066 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.126:39339)
I20260927 15:24:04.735013 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.719376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.2.246.126:39339)
0927 15:24:04.719610 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.719616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.719637 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:24:04.719761 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.719768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.719856 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.719983 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:04.721574 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.722441 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.723951 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.724409 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.724574 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.727724 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:04.727729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:04.727732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:04.727773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:04.731205 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.732537 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.732543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.732546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.732722 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:04.733188 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:04.733194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:04.733197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:04.733625 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:04.733980 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.734328 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.734635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:24:04.739621 19143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:24:04.748649 19143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:24:04.788235 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.766213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.2.246.66:44371)
0927 15:24:04.776722 (+ 10509us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.776737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.777889 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 15:24:04.777997 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.778006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.778297 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.778517 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:04.779676 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.781356 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.785280 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.786784 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.786969 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.787192 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.787679 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.787852 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10283,"thread_start_us":10123,"threads_started":1}
I20260927 15:24:04.788622 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.766543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44371 (local address 127.0.0.1:33360)
0927 15:24:04.777340 (+ 10797us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.777365 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.777566 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.778820 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.778828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.778873 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:04.779450 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.779467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.781552 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.781560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.785058 (+ 3498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.785084 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.787064 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.787070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.787378 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.787423 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.787606 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.787844 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10640,"mutex_wait_us":8,"spinlock_wait_cycles":896,"thread_start_us":10441,"threads_started":1}
I20260927 15:24:04.789424 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.765603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45940)
0927 15:24:04.766579 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.766636 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.766856 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.778820 (+ 11964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.778828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.778879 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:04.780140 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.780153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.782123 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.782131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.785650 (+ 3519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.785670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.788440 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.788447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.788640 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.788680 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.788783 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.788957 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":1311,"spinlock_wait_cycles":8320,"thread_start_us":523,"threads_started":1}
I20260927 15:24:04.790764 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.766005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.2.246.65:45785)
0927 15:24:04.777809 (+ 11804us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.777816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.777882 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:24:04.777997 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.778006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.778298 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.778517 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:04.780371 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.781942 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.785966 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.787793 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.787966 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.788268 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.790146 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.790390 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11599,"thread_start_us":11634,"threads_started":1}
I20260927 15:24:04.791689 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.767586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36545 (local address 127.0.0.1:35616)
0927 15:24:04.780312 (+ 12726us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.780340 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.780546 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.781596 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.781600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.781634 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:04.782462 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.782491 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.785085 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.785091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.787734 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.787752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.790793 (+ 3041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.790801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.790998 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.791034 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.791155 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.791313 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12552,"thread_start_us":12328,"threads_started":1}
I20260927 15:24:04.791815 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.767733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.2.246.67:36545)
0927 15:24:04.779501 (+ 11768us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.779509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.780703 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 15:24:04.780840 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.780848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.781232 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.781613 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:04.782801 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.784889 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.788022 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.790117 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.790257 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.790523 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.791226 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.791434 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11420,"thread_start_us":11259,"threads_started":1}
I20260927 15:24:04.792987 19395 tablet_service.cc:1467] Processing CreateTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 (DEFAULT_TABLE table=test-table [id=8a3dd4b9ff784c5a97608a3dbbe23d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:04.794552 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b2bc3a65224bb0a71540d1f11fe973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:04.794389 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 (DEFAULT_TABLE table=test-table [id=8a3dd4b9ff784c5a97608a3dbbe23d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:04.795799 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b2bc3a65224bb0a71540d1f11fe973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:04.796892 19527 tablet_service.cc:1467] Processing CreateTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 (DEFAULT_TABLE table=test-table [id=8a3dd4b9ff784c5a97608a3dbbe23d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:04.798862 19527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b2bc3a65224bb0a71540d1f11fe973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:04.814656 19608 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Bootstrap starting.
I20260927 15:24:04.816035 19609 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Bootstrap starting.
I20260927 15:24:04.819444 19608 tablet_bootstrap.cc:654] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:04.821215 19608 log.cc:824] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:04.821426 19609 tablet_bootstrap.cc:654] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:04.822090 19610 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Bootstrap starting.
I20260927 15:24:04.822829 19609 log.cc:824] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:04.824940 19608 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: No bootstrap required, opened a new log
I20260927 15:24:04.825274 19608 ts_tablet_manager.cc:1397] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260927 15:24:04.825563 19609 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: No bootstrap required, opened a new log
I20260927 15:24:04.825819 19609 ts_tablet_manager.cc:1397] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 15:24:04.826043 19610 tablet_bootstrap.cc:654] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:04.827348 19610 log.cc:824] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:04.832069 19610 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: No bootstrap required, opened a new log
I20260927 15:24:04.832526 19610 ts_tablet_manager.cc:1397] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 15:24:04.835044 19608 raft_consensus.cc:348] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.835505 19608 raft_consensus.cc:374] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:04.835601 19608 raft_consensus.cc:729] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebfbbd8289f4d2d8826cd31951fb5b4, State: Initialized, Role: FOLLOWER
I20260927 15:24:04.836462 19608 consensus_queue.cc:260] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.837764 19460 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
I20260927 15:24:04.838605 19608 ts_tablet_manager.cc:1428] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260927 15:24:04.838433 19609 raft_consensus.cc:348] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.839082 19609 raft_consensus.cc:374] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:04.839218 19609 raft_consensus.cc:729] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5da2fe0dd24d1d8829d4c9c0673d48, State: Initialized, Role: FOLLOWER
I20260927 15:24:04.840379 19609 consensus_queue.cc:260] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.843357 19609 ts_tablet_manager.cc:1428] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent starting tablet: real 0.017s user 0.005s sys 0.011s
I20260927 15:24:04.847138 19610 raft_consensus.cc:348] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.848132 19610 raft_consensus.cc:374] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:04.848277 19610 raft_consensus.cc:729] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eaa55af4cc94abc8db42239a2c2987f, State: Initialized, Role: FOLLOWER
I20260927 15:24:04.849195 19610 consensus_queue.cc:260] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.850853 19592 heartbeater.cc:503] Master 127.2.246.126:39339 was elected leader, sending a full tablet report...
I20260927 15:24:04.851369 19610 ts_tablet_manager.cc:1428] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
W20260927 15:24:04.902631 19461 tablet.cc:2369] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:24:04.926541 19593 tablet.cc:2369] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:04.966962 19616 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:04.967365 19616 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.969796 19616 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785), 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371)
I20260927 15:24:04.977583 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.970431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:36111 (local address 127.2.246.66:44371)
0927 15:24:04.970706 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.970715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.970943 (+ 228us) server_negotiation.cc:407] Connection header received
0927 15:24:04.971223 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.971229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.971329 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.971490 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:04.971996 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.973130 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.975695 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.976558 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.976748 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.976829 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.977165 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.977302 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:24:04.977631 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.970352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44371 (local address 127.2.246.67:36111)
0927 15:24:04.970805 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.970826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.971013 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.971543 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.971551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.971583 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:04.971829 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.971841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.973327 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.973336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.975450 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.975465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.976749 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.976756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.976988 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.977020 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.977137 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.977295 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":116}
I20260927 15:24:04.978065 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.970603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.2.246.67:38781)
0927 15:24:04.971515 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:04.971534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:04.971673 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:04.972056 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:04.972059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:04.972078 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:04.972378 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.972386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.973856 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.973864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.975662 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:04.975678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.976927 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.976934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:04.977156 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.977181 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:24:04.977295 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:24:04.977450 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":34,"thread_start_us":555,"threads_started":1}
I20260927 15:24:04.978219 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:04.970509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:38781 (local address 127.2.246.65:45785)
0927 15:24:04.970802 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:04.970808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:04.971659 (+ 851us) server_negotiation.cc:407] Connection header received
0927 15:24:04.971823 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:04.971828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:04.971921 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:04.972072 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:04.972533 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.973643 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.975837 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:04.976693 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:04.976903 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:04.976999 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:04.977323 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 15:24:04.977467 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 15:24:04.978750 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b2bc3a65224bb0a71540d1f11fe973" candidate_uuid: "7eaa55af4cc94abc8db42239a2c2987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" is_pre_election: true
I20260927 15:24:04.979300 19415 raft_consensus.cc:2454] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eaa55af4cc94abc8db42239a2c2987f in term 0.
I20260927 15:24:04.979419 19283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b2bc3a65224bb0a71540d1f11fe973" candidate_uuid: "7eaa55af4cc94abc8db42239a2c2987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" is_pre_election: true
I20260927 15:24:04.980026 19480 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4; no voters:
I20260927 15:24:04.980051 19283 raft_consensus.cc:2454] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eaa55af4cc94abc8db42239a2c2987f in term 0.
I20260927 15:24:04.980458 19616 raft_consensus.cc:2781] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:04.980618 19616 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:04.980760 19616 raft_consensus.cc:3037] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:04.983073 19616 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.983800 19616 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [CANDIDATE]: Term 1 election: Requested vote from peers fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785), 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371)
I20260927 15:24:04.984452 19283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b2bc3a65224bb0a71540d1f11fe973" candidate_uuid: "7eaa55af4cc94abc8db42239a2c2987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
I20260927 15:24:04.984421 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b2bc3a65224bb0a71540d1f11fe973" candidate_uuid: "7eaa55af4cc94abc8db42239a2c2987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
I20260927 15:24:04.984742 19283 raft_consensus.cc:3037] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:04.984750 19415 raft_consensus.cc:3037] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:04.987589 19283 raft_consensus.cc:2454] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eaa55af4cc94abc8db42239a2c2987f in term 1.
I20260927 15:24:04.988092 19482 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eaa55af4cc94abc8db42239a2c2987f, fe5da2fe0dd24d1d8829d4c9c0673d48; no voters:
I20260927 15:24:04.988477 19616 raft_consensus.cc:2781] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:04.988529 19415 raft_consensus.cc:2454] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eaa55af4cc94abc8db42239a2c2987f in term 1.
I20260927 15:24:04.989168 19616 raft_consensus.cc:686] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 LEADER]: Becoming Leader. State: Replica: 7eaa55af4cc94abc8db42239a2c2987f, State: Running, Role: LEADER
I20260927 15:24:04.990056 19616 consensus_queue.cc:237] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:04.995759 19143 catalog_manager.cc:5495] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67). New cstate: current_term: 1 leader_uuid: "7eaa55af4cc94abc8db42239a2c2987f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } health_report { overall_health: HEALTHY } } }
I20260927 15:24:05.049533 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.042874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.2.246.67:36545)
0927 15:24:05.043195 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:05.043200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:05.043299 (+ 99us) server_negotiation.cc:407] Connection header received
0927 15:24:05.043613 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:05.043617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:05.043771 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:05.043937 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:05.044401 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.045223 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.046832 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.047229 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.047333 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:05.048028 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:05.048825 (+ 797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:24:05.048947 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:05.049176 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:24:05.049256 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 15:24:05.060645 19283 raft_consensus.cc:1261] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Refusing update from remote peer 7eaa55af4cc94abc8db42239a2c2987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:05.060716 19415 raft_consensus.cc:1261] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Refusing update from remote peer 7eaa55af4cc94abc8db42239a2c2987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:05.062235 19618 consensus_queue.cc:1035] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:05.062803 19616 consensus_queue.cc:1035] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:05.096638 19142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:05.097028 19142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:05.102202 19142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Sending DeleteTablet for 3 replicas of tablet 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:05.103726 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:05 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:45940
I20260927 15:24:05.103669 19527 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:05 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:35616
I20260927 15:24:05.104008 19395 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b2bc3a65224bb0a71540d1f11fe973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:05 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:33360
I20260927 15:24:05.105608 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19180
I20260927 15:24:05.116122 19631 tablet_replica.cc:330] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: stopping tablet replica
I20260927 15:24:05.116708 19631 raft_consensus.cc:2229] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:24:05.117547 19631 raft_consensus.cc:2258] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:05.118368 19633 tablet_replica.cc:330] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: stopping tablet replica
I20260927 15:24:05.119073 19633 raft_consensus.cc:2229] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:05.119815 19633 raft_consensus.cc:2258] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:05.120512 19631 ts_tablet_manager.cc:1905] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:05.123267 19633 ts_tablet_manager.cc:1905] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:05.129145 19632 tablet_replica.cc:330] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: stopping tablet replica
I20260927 15:24:05.129573 19631 ts_tablet_manager.cc:1918] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:05.129658 19632 raft_consensus.cc:2229] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:05.129799 19631 log.cc:1196] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:05.130288 19632 raft_consensus.cc:2258] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:05.130455 19631 ts_tablet_manager.cc:1939] T 85b2bc3a65224bb0a71540d1f11fe973 P 7eaa55af4cc94abc8db42239a2c2987f: Deleting consensus metadata
I20260927 15:24:05.132699 19129 catalog_manager.cc:4841] TS 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): tablet 85b2bc3a65224bb0a71540d1f11fe973 (table test-table [id=8a3dd4b9ff784c5a97608a3dbbe23d1f]) successfully deleted
I20260927 15:24:05.133109 19633 ts_tablet_manager.cc:1918] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:05.133371 19633 log.cc:1196] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:05.133970 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.106893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.2.246.65:45785)
0927 15:24:05.107127 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:05.107135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:05.107165 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:24:05.107454 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:05.107463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:05.107575 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:05.107722 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:05.110517 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.112085 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.115022 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.115866 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.116001 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:05.122870 (+ 6869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:05.122896 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:05.122917 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:05.122983 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:05.127234 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.128380 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.128392 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.128395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.128569 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.131140 (+ 2571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.131145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.131148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.131684 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:05.132000 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:05.133393 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0927 15:24:05.133631 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:24:05.134517 19633 ts_tablet_manager.cc:1939] T 85b2bc3a65224bb0a71540d1f11fe973 P 8ebfbbd8289f4d2d8826cd31951fb5b4: Deleting consensus metadata
I20260927 15:24:05.134764 19323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:05.137552 19127 catalog_manager.cc:4841] TS 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): tablet 85b2bc3a65224bb0a71540d1f11fe973 (table test-table [id=8a3dd4b9ff784c5a97608a3dbbe23d1f]) successfully deleted
I20260927 15:24:05.233739 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19180
I20260927 15:24:05.253654 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19332
W20260927 15:24:05.253957 19129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:45785 (error 108)
I20260927 15:24:05.257292 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.256560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45962)
0927 15:24:05.256826 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.257045 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:24:05.257875 19129 catalog_manager.cc:4584] TS fe5da2fe0dd24d1d8829d4c9c0673d48 (127.2.246.65:45785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85b2bc3a65224bb0a71540d1f11fe973: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
I20260927 15:24:05.273896 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.255047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.2.246.66:44371)
0927 15:24:05.255313 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:05.255321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:05.255353 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:24:05.255619 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:05.255625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:05.255732 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:05.255833 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:05.257654 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.259035 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.260746 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.261470 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.261637 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:05.264761 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:05.264793 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:05.264821 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:05.264905 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:05.269824 (+ 4919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.270936 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.270949 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.270953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.271190 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.271653 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.271658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.271662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.272308 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:05.272682 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:05.273058 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 15:24:05.273358 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 15:24:05.275049 19455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:05.312160 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.311359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45964)
0927 15:24:05.311716 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.311909 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 15:24:05.313921 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.313336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45966)
0927 15:24:05.313566 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.313705 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 15:24:05.370715 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19332
I20260927 15:24:05.374969 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.374134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45968)
0927 15:24:05.374502 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.374675 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 15:24:05.376794 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.376170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45984)
0927 15:24:05.376430 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.376566 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:24:05.390653 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19464
I20260927 15:24:05.413410 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.391738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.2.246.67:36545)
0927 15:24:05.391935 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:05.391940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:05.392073 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:24:05.392385 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:05.392390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:05.392474 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:05.392603 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:05.395220 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.396164 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.398929 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:05.399741 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:05.399998 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:05.404953 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:05.405027 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:05.405051 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:05.405113 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:05.409212 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.410704 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.410717 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.410719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.410922 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:05.411482 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:05.411487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:05.411489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:05.411976 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:05.412271 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:05.412827 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:24:05.413052 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:24:05.414229 19587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:05.457970 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.457384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:45994)
0927 15:24:05.457621 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.457710 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:24:05.459167 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.458693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:46000)
0927 15:24:05.458911 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.458979 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 15:24:05.559124 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19464
I20260927 15:24:05.580739 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19110
I20260927 15:24:05.582350 19173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:05.723455 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:05.722917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45785 (local address 127.0.0.1:46006)
0927 15:24:05.723143 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:05.723251 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:24:05.723439 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19110
I20260927 15:24:25.751332 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--webserver_interface=127.2.246.126
--webserver_port=40493
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:39339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:26.102588 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:26.102962 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:26.103123 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:26.111204 19639 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:24:26.111357 19639 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:24:26.111459 19639 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:24:26.111557 19639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:26.111666 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:26.111775 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:26.111845 19639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:26.111923 19639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:26.122427 19639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:39339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:39339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=40493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:26.123916 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:26.125334 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:26.140137 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:26.140280 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:26.140223 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:26.140590 19639 server_base.cc:1034] running on GCE node
I20260927 15:24:26.142304 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:26.144608 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:26.146023 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790522666145925 us; error 55 us; skew 500 ppm
I20260927 15:24:26.146559 19639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:26 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:26.151000 19639 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:24:26.153213 19639 webserver.cc:460] Webserver started at http://127.2.246.126:40493/ using document root <none> and password file <none>
I20260927 15:24:26.154307 19639 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:26.154429 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:26.162082 19639 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:24:26.166427 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:26.168336 19639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:24:26.168638 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "9396e5648b40465f98ca4f5f943a26fb"
format_stamp: "Formatted at 2026-09-27 15:23:39 on dist-test-slave-14jq"
server_key: "c1fc4a432fc4a34715e55d050f31644b"
server_key_iv: "fa143b7dc6a72d9a9c540f2f0f54c6fe"
server_key_version: "encryptionkey@0"
I20260927 15:24:26.170287 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:26.191121 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:26.197639 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:26.198143 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:26.223641 19639 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:39339
I20260927 15:24:26.223699 19705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:39339 every 8 connection(s)
I20260927 15:24:26.225872 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:26.233270 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19639
I20260927 15:24:26.234311 19706 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:24:26.234368 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:45785
--local_ip_for_outbound_sockets=127.2.246.65
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=36257
--webserver_interface=127.2.246.65
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:24:26.238020 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap starting.
I20260927 15:24:26.267102 19706 log.cc:824] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:26.290323 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:24:26.291023 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Bootstrap complete.
I20260927 15:24:26.305348 19706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:26.307220 19706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Initialized, Role: FOLLOWER
I20260927 15:24:26.308308 19706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:26.308740 19706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:26.308917 19706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:26.309176 19706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:24:26.315559 19706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:26.316282 19706 leader_election.cc:302] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9396e5648b40465f98ca4f5f943a26fb; no voters:
I20260927 15:24:26.317116 19706 leader_election.cc:288] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:24:26.317454 19710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:24:26.318886 19710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [term 3 LEADER]: Becoming Leader. State: Replica: 9396e5648b40465f98ca4f5f943a26fb, State: Running, Role: LEADER
I20260927 15:24:26.319489 19706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:26.319852 19710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } }
I20260927 15:24:26.324251 19711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:24:26.324683 19711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:26.325168 19712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9396e5648b40465f98ca4f5f943a26fb. Latest consensus state: current_term: 3 leader_uuid: "9396e5648b40465f98ca4f5f943a26fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9396e5648b40465f98ca4f5f943a26fb" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 39339 } } }
I20260927 15:24:26.325470 19712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:26.328665 19718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:26.340976 19718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:26.342998 19718 catalog_manager.cc:1259] Loaded cluster ID: 32aabec4e8594b428cf9f7b9dc4bf40a
I20260927 15:24:26.343175 19718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:26.349603 19718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:26.352288 19718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Loaded TSK: 0
I20260927 15:24:26.352519 19718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9396e5648b40465f98ca4f5f943a26fb: Loaded TSK: 1
F20260927 15:24:26.354377 19718 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f42ff6ea9ad google::LogMessage::Fail() at ??:0
@ 0x7f42ff6ed88a google::LogMessage::SendToLog() at ??:0
@ 0x7f42ff6ea4de google::LogMessage::Flush() at ??:0
@ 0x7f42ff6eba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f430344e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f430343d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f431511d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f431511c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f43151a82a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f43151a80c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f43151f25e9 std::function<>::operator()() at ??:0
@ 0x7f4315120cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f431511ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43151a6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f43151a6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4302336345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f430233991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4302339799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4302314eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43055ee6db start_thread at ??:0
@ 0x7f42fe58461f clone at ??:0
W20260927 15:24:26.710284 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:26.710593 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:26.710714 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:26.718566 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:26.718708 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:26.718830 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:26.729198 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:45785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=36257
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:26.730816 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:26.732594 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:26.748206 19753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:26.748337 19754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:26.748772 19756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:26.755002 19708 server_base.cc:1034] running on GCE node
I20260927 15:24:26.755980 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:26.757354 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:26.758872 19708 hybrid_clock.cc:648] HybridClock initialized: now 1790522666758756 us; error 106 us; skew 500 ppm
I20260927 15:24:26.759320 19708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:26 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:26.763278 19708 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:24:26.765326 19708 webserver.cc:460] Webserver started at http://127.2.246.65:36257/ using document root <none> and password file <none>
I20260927 15:24:26.766217 19708 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:26.766383 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:26.772714 19708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 15:24:26.776492 19763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:26.778424 19708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:24:26.778658 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "3471a8661d89199214e0732f2bd10933"
server_key_iv: "4ac6b33e1ea902fc8c3d60f4db389445"
server_key_version: "encryptionkey@0"
I20260927 15:24:26.779697 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:26.800359 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:26.806617 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:26.806977 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:26.808234 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:26.810544 19770 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 15:24:26.818493 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:24:26.818735 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 15:24:26.818909 19708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:24:26.822755 19708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:24:26.822903 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260927 15:24:26.823607 19770 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Bootstrap starting.
I20260927 15:24:26.836339 19770 log.cc:824] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:26.857213 19770 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:24:26.858906 19770 tablet_bootstrap.cc:492] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Bootstrap complete.
I20260927 15:24:26.859692 19770 ts_tablet_manager.cc:1397] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.004s
I20260927 15:24:26.872613 19770 raft_consensus.cc:348] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:26.874809 19770 raft_consensus.cc:729] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5da2fe0dd24d1d8829d4c9c0673d48, State: Initialized, Role: FOLLOWER
I20260927 15:24:26.875756 19770 consensus_queue.cc:260] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:26.878019 19770 ts_tablet_manager.cc:1428] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48: Time spent starting tablet: real 0.018s user 0.009s sys 0.010s
I20260927 15:24:26.880936 19708 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45785
I20260927 15:24:26.881075 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45785 every 8 connection(s)
I20260927 15:24:26.882632 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:26.883270 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19708
I20260927 15:24:26.884402 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:44371
--local_ip_for_outbound_sockets=127.2.246.66
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=39205
--webserver_interface=127.2.246.66
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:24:26.889750 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:26.886563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:55543)
0927 15:24:26.887810 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:26.888189 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":463,"threads_started":1}
W20260927 15:24:26.890648 19879 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:39339 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:39339: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
I20260927 15:24:26.891932 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:26.891323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:45323)
0927 15:24:26.891583 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:26.891727 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:24:26.893288 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:26.892728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:36047)
0927 15:24:26.892960 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:26.893065 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 15:24:26.893692 19879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:24:27.322707 19882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:27.323129 19882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:27.323318 19882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:27.332770 19882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:27.332954 19882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:27.333177 19882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:27.346043 19882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:44371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=39205
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:27.348088 19882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:27.350073 19882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:27.367623 19890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:27.367785 19888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:27.368415 19887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:27.368808 19882 server_base.cc:1034] running on GCE node
I20260927 15:24:27.369560 19882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:27.370924 19882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:27.372184 19882 hybrid_clock.cc:648] HybridClock initialized: now 1790522667372109 us; error 56 us; skew 500 ppm
I20260927 15:24:27.372578 19882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:27 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522667, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:27.377425 19882 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:24:27.379648 19882 webserver.cc:460] Webserver started at http://127.2.246.66:39205/ using document root <none> and password file <none>
I20260927 15:24:27.380537 19882 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:27.380760 19882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:27.387988 19882 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:24:27.392216 19897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:27.394356 19882 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:24:27.394624 19882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4"
format_stamp: "Formatted at 2026-09-27 15:23:40 on dist-test-slave-14jq"
server_key: "8a3c45bdf784593445eca8e4105ee52a"
server_key_iv: "5f81308a026dc8e50d55b4b954fff0a9"
server_key_version: "encryptionkey@0"
I20260927 15:24:27.395792 19882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:27.415119 19882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:27.420179 19882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:27.420495 19882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:27.421787 19882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:27.423377 19882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:27.423511 19882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:27.423677 19882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:27.423774 19882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:27.467360 19882 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:44371
I20260927 15:24:27.467407 20009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:44371 every 8 connection(s)
I20260927 15:24:27.468966 19882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:27.473026 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 19882
I20260927 15:24:27.474093 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:36545
--local_ip_for_outbound_sockets=127.2.246.67
--tserver_master_addrs=127.2.246.126:39339
--webserver_port=41635
--webserver_interface=127.2.246.67
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:24:27.475311 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:27.472980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:55117)
0927 15:24:27.474049 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:27.474357 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":497,"threads_started":1}
W20260927 15:24:27.476151 20010 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:39339 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:39339: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
I20260927 15:24:27.477290 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:27.476917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:50567)
0927 15:24:27.477099 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:27.477176 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 15:24:27.478344 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:27.478008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:56965)
0927 15:24:27.478145 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:27.478240 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:24:27.478720 20010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:24:27.895268 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:27.894379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:54047)
0927 15:24:27.894904 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:27.894997 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
W20260927 15:24:27.897050 20013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:27.897600 20013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:27.897816 20013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:27.907567 20013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:27.907793 20013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:27.908008 20013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:27.920799 20013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=41635
--tserver_master_addrs=127.2.246.126:39339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:27.922680 20013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:27.924322 20013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:27.940563 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:27.940754 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:27.941406 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:27.942093 20013 server_base.cc:1034] running on GCE node
I20260927 15:24:27.942950 20013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:27.944253 20013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:27.945518 20013 hybrid_clock.cc:648] HybridClock initialized: now 1790522667945422 us; error 79 us; skew 500 ppm
I20260927 15:24:27.946038 20013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:27 dist-test-slave-14jq krb5kdc[18557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522667, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:27.950587 20013 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:24:27.952920 20013 webserver.cc:460] Webserver started at http://127.2.246.67:41635/ using document root <none> and password file <none>
I20260927 15:24:27.954120 20013 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:27.954309 20013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:27.962366 20013 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:24:27.966289 20029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:27.968300 20013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:24:27.968575 20013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "7eaa55af4cc94abc8db42239a2c2987f"
format_stamp: "Formatted at 2026-09-27 15:23:41 on dist-test-slave-14jq"
server_key: "6cf01f1c77c307f0f598a504186f91b8"
server_key_iv: "68932e6afa51724ab2af1c1811113a0f"
server_key_version: "encryptionkey@0"
I20260927 15:24:27.969972 20013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:27.987963 20013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:27.994093 20013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:27.994465 20013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:27.995828 20013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:27.997704 20013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:27.997810 20013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:27.998039 20013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:27.998143 20013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:28.042831 20013 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:36545
I20260927 15:24:28.042883 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:36545 every 8 connection(s)
I20260927 15:24:28.044718 20013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:28.049225 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20013
I20260927 15:24:28.051908 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.048675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:55685)
0927 15:24:28.050058 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.050479 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":493,"threads_started":1}
W20260927 15:24:28.052884 20142 heartbeater.cc:644] Failed to heartbeat to 127.2.246.126:39339 (0 consecutive failures): Network error: Failed to ping master at 127.2.246.126:39339: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
I20260927 15:24:28.054134 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.053628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:46063)
0927 15:24:28.053912 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.054012 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 15:24:28.055245 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.054878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:57585)
0927 15:24:28.055029 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.055127 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:24:28.055543 20142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:24:28.441552 20146 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:28.441996 20146 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:28.443567 20146 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
Sep 27 15:24:28 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
Sep 27 15:24:28 dist-test-slave-14jq krb5kdc[18557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:24:28.480751 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.479313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:39387)
0927 15:24:28.480041 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.480315 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":293,"threads_started":1}
I20260927 15:24:28.489504 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.444293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44371 (local address 127.2.246.65:48289)
0927 15:24:28.457104 (+ 12811us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.457143 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:28.457389 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:28.461204 (+ 3815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:28.461211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:28.462308 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:28.463227 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:28.463250 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.465229 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.465243 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:28.467720 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:28.467748 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.469400 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.469409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:28.469868 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:28.475476 (+ 5608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:28.475503 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:28.479313 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:28.483704 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:28.483715 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:28.483734 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:28.484472 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:28.485274 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:28.485279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:28.485282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:28.485610 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:28.487052 (+ 1442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:28.487063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:28.487830 (+ 767us) client_negotiation.cc:769] Sending connection context
0927 15:24:28.488209 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 15:24:28.489036 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12597,"mutex_wait_us":567,"spinlock_wait_cycles":36608,"thread_start_us":11505,"threads_started":1}
I20260927 15:24:28.489743 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.444543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:51205 (local address 127.2.246.67:36545)
0927 15:24:28.458475 (+ 13932us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:28.458488 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:28.458531 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:24:28.460133 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:28.460143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:28.460587 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:28.461000 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:28.463474 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.464800 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.467892 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.469093 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.469340 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:28.479989 (+ 10649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:28.480004 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:28.480034 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:28.480102 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:28.484017 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:28.484903 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:28.484911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:28.484918 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:28.485151 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:28.485796 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:28.485803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:28.485807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:28.486773 (+ 966us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:28.487013 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:28.488628 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0927 15:24:28.489229 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13757,"thread_start_us":13551,"threads_started":1}
I20260927 15:24:28.489529 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.444117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:36545 (local address 127.2.246.65:51205)
0927 15:24:28.456204 (+ 12087us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.456271 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:28.457114 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:28.461168 (+ 4054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:28.461181 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:28.462322 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:28.463207 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:28.463232 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.465050 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.465055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:28.467680 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:28.467702 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.469759 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.469766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:28.469999 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:28.475202 (+ 5203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:28.475236 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:28.479434 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:28.484194 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:28.484200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:28.484203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:28.484730 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:28.485274 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:28.485279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:28.485282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:28.485610 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:28.487052 (+ 1442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:28.487063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:28.487800 (+ 737us) client_negotiation.cc:769] Sending connection context
0927 15:24:28.488209 (+ 409us) client_negotiation.cc:240] Negotiation successful
0927 15:24:28.488883 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11857,"mutex_wait_us":128,"spinlock_wait_cycles":42496,"thread_start_us":13439,"threads_started":1}
I20260927 15:24:28.491024 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.444483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:48289 (local address 127.2.246.66:44371)
0927 15:24:28.458327 (+ 13844us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:28.458344 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:28.458392 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:24:28.460214 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:28.460227 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:28.460770 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:28.461230 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:28.463579 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.464974 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.468012 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:28.469093 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:28.469341 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:28.479861 (+ 10520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:28.479874 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:28.479890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:28.479936 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:28.483470 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:28.484698 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:28.484708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:28.484716 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:28.485110 (+ 394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:28.485797 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:28.485803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:28.485806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:28.486773 (+ 967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:28.487016 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:28.488632 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0927 15:24:28.490684 (+ 2052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13621,"thread_start_us":13470,"threads_started":1}
W20260927 15:24:28.493922 19766 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
W20260927 15:24:28.494292 19764 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:28.494472 19764 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe5da2fe0dd24d1d8829d4c9c0673d48; no voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4
I20260927 15:24:28.494928 20146 raft_consensus.cc:2726] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:24:28.897082 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:28.896520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:54137)
0927 15:24:28.896769 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:28.896899 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 15:24:29.057545 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:29.056439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:41809)
0927 15:24:29.057118 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:29.057279 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":307,"threads_started":1}
I20260927 15:24:29.483124 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:29.482182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:55537)
0927 15:24:29.482826 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:29.482949 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":330,"threads_started":1}
I20260927 15:24:29.899430 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:29.898318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:46747)
0927 15:24:29.898964 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:29.899087 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":305,"threads_started":1}
I20260927 15:24:30.059948 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:30.058915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:46133)
0927 15:24:30.059482 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:30.059589 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":306,"threads_started":1}
I20260927 15:24:30.439728 20156 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:30.440022 20156 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:30.440891 20156 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
W20260927 15:24:30.442438 19764 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
W20260927 15:24:30.442795 19766 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:30.442975 19766 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe5da2fe0dd24d1d8829d4c9c0673d48; no voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4
I20260927 15:24:30.443460 20156 raft_consensus.cc:2726] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:24:30.485711 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:30.484553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:55715)
0927 15:24:30.485344 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:30.485480 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":455,"threads_started":1}
I20260927 15:24:30.902084 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:30.900798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:35177)
0927 15:24:30.901617 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:30.901751 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":424,"threads_started":1}
I20260927 15:24:31.062479 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:31.061490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:47839)
0927 15:24:31.062180 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:31.062307 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":418,"threads_started":1}
I20260927 15:24:31.487972 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:31.487071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:34753)
0927 15:24:31.487646 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:31.487759 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260927 15:24:31.904742 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:31.903576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:53907)
0927 15:24:31.904342 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:31.904508 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":383,"threads_started":1}
I20260927 15:24:32.064802 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:32.063772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:60641)
0927 15:24:32.064450 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:32.064563 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":417,"threads_started":1}
I20260927 15:24:32.490442 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:32.489331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:45819)
0927 15:24:32.490086 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:32.490211 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":344,"threads_started":1}
I20260927 15:24:32.736227 20164 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:32.736446 20164 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:32.737329 20164 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
W20260927 15:24:32.738775 19764 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
W20260927 15:24:32.739094 19766 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:32.739319 19766 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe5da2fe0dd24d1d8829d4c9c0673d48; no voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4
I20260927 15:24:32.739657 20164 raft_consensus.cc:2726] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:24:32.907373 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:32.906191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:54887)
0927 15:24:32.906991 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:32.907128 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":416,"threads_started":1}
I20260927 15:24:33.067651 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:33.066433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:41687)
0927 15:24:33.067233 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:33.067376 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":389,"threads_started":1}
I20260927 15:24:33.493103 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:33.491801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:49149)
0927 15:24:33.492666 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:33.492828 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":452,"threads_started":1}
I20260927 15:24:33.910193 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:33.908801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:38103)
0927 15:24:33.909687 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:33.909885 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":442,"threads_started":1}
I20260927 15:24:34.070182 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:34.069107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:59651)
0927 15:24:34.069765 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:34.069961 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":355,"threads_started":1}
I20260927 15:24:34.495847 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:34.494601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:36037)
0927 15:24:34.495311 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:34.495517 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":315,"threads_started":1}
I20260927 15:24:34.912555 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:34.911496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:36483)
0927 15:24:34.912177 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:34.912299 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":366,"threads_started":1}
I20260927 15:24:35.072734 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:35.071493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:45577)
0927 15:24:35.072308 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:35.072451 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":419,"threads_started":1}
I20260927 15:24:35.498310 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:35.497277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:53315)
0927 15:24:35.497969 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:35.498086 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":294,"threads_started":1}
I20260927 15:24:35.915237 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:35.914107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:60939)
0927 15:24:35.914900 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:35.915034 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":370,"threads_started":1}
I20260927 15:24:36.075302 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:36.074240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:54711)
0927 15:24:36.074924 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:36.075053 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":327,"threads_started":1}
I20260927 15:24:36.500769 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:36.499572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:54059)
0927 15:24:36.500342 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:36.500513 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":361,"threads_started":1}
I20260927 15:24:36.917514 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:36.916577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:35301)
0927 15:24:36.917206 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:36.917326 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":348,"threads_started":1}
I20260927 15:24:37.077657 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:37.076600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:59203)
0927 15:24:37.077330 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:37.077463 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":373,"threads_started":1}
I20260927 15:24:37.458640 20179 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:37.458845 20179 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:37.459594 20179 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
W20260927 15:24:37.460894 19766 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
W20260927 15:24:37.461158 19764 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:37.461370 19764 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe5da2fe0dd24d1d8829d4c9c0673d48; no voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4
I20260927 15:24:37.461678 20179 raft_consensus.cc:2726] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:24:37.503058 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:37.502173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:48089)
0927 15:24:37.502742 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:37.502853 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":320,"threads_started":1}
I20260927 15:24:37.920234 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:37.919006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:38071)
0927 15:24:37.919777 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:37.919961 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":365,"threads_started":1}
I20260927 15:24:38.080111 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:38.079171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:59011)
0927 15:24:38.079797 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:38.079908 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260927 15:24:38.505283 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:38.504250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:53941)
0927 15:24:38.505010 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:38.505108 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
I20260927 15:24:38.922708 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:38.921419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:39785)
0927 15:24:38.922334 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:38.922479 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":488,"threads_started":1}
I20260927 15:24:39.082386 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:39.081215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:60523)
0927 15:24:39.082037 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:39.082169 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":356,"threads_started":1}
I20260927 15:24:39.507807 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:39.506755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:55121)
0927 15:24:39.507433 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:39.507572 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":389,"threads_started":1}
I20260927 15:24:39.925199 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:39.924137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:56671)
0927 15:24:39.924809 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:39.924953 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":358,"threads_started":1}
I20260927 15:24:40.084789 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:40.083628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:37167)
0927 15:24:40.084432 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:40.084559 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":446,"threads_started":1}
I20260927 15:24:40.509996 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:40.509052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:57349)
0927 15:24:40.509679 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:40.509799 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
I20260927 15:24:40.927522 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:40.926558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:39429)
0927 15:24:40.927178 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:40.927327 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":353,"threads_started":1}
I20260927 15:24:41.087628 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:41.086378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:55931)
0927 15:24:41.087164 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:41.087318 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":388,"threads_started":1}
I20260927 15:24:41.512281 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:41.511019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:47805)
0927 15:24:41.511790 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:41.511947 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260927 15:24:41.930248 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:41.929152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:39681)
0927 15:24:41.929828 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:41.930019 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
I20260927 15:24:42.090379 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:42.089163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:48129)
0927 15:24:42.089952 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:42.090098 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":357,"threads_started":1}
I20260927 15:24:42.261530 20195 raft_consensus.cc:482] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:42.261783 20195 raft_consensus.cc:504] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe5da2fe0dd24d1d8829d4c9c0673d48" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45785 } } peers { permanent_uuid: "8ebfbbd8289f4d2d8826cd31951fb5b4" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44371 } } peers { permanent_uuid: "7eaa55af4cc94abc8db42239a2c2987f" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 36545 } }
I20260927 15:24:42.262761 20195 leader_election.cc:288] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371), 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545)
W20260927 15:24:42.264045 19764 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebfbbd8289f4d2d8826cd31951fb5b4 (127.2.246.66:44371): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
W20260927 15:24:42.264289 19766 leader_election.cc:341] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7eaa55af4cc94abc8db42239a2c2987f (127.2.246.67:36545): Not found: Tablet not found: 85b2bc3a65224bb0a71540d1f11fe973
I20260927 15:24:42.264480 19766 leader_election.cc:302] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe5da2fe0dd24d1d8829d4c9c0673d48; no voters: 7eaa55af4cc94abc8db42239a2c2987f, 8ebfbbd8289f4d2d8826cd31951fb5b4
I20260927 15:24:42.264832 20195 raft_consensus.cc:2726] T 85b2bc3a65224bb0a71540d1f11fe973 P fe5da2fe0dd24d1d8829d4c9c0673d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:24:42.515801 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:42.514132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.66:41961)
0927 15:24:42.515337 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:42.515519 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":811,"threads_started":1}
I20260927 15:24:42.932622 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:42.931634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.65:46469)
0927 15:24:42.932324 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:42.932442 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":329,"threads_started":1}
I20260927 15:24:43.053586 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19708
I20260927 15:24:43.077627 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:43.055953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.2.246.65:45785)
0927 15:24:43.056758 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:43.056771 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:43.056809 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:24:43.056906 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:43.056918 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:43.057241 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:43.057440 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:43.059230 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.060374 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.062092 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.062757 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.062923 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:43.070794 (+ 7871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:43.070820 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:43.070840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:43.070886 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:43.074382 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.075282 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.075291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.075294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.075443 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.075862 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.075867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.075871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.076295 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:43.076616 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:43.076970 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 15:24:43.077262 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":330,"threads_started":1}
I20260927 15:24:43.079468 19873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:43.093279 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:43.092188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:39339 (local address 127.2.246.67:33119)
0927 15:24:43.092938 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:43.093090 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.126:39339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":362,"threads_started":1}
I20260927 15:24:43.184360 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19708
I20260927 15:24:43.207556 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 19882
I20260927 15:24:43.234826 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:43.208742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.2.246.66:44371)
0927 15:24:43.209382 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:43.209388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:43.209410 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:24:43.209491 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:43.209495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:43.209575 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:43.209733 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:43.211563 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.212511 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.214039 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.214551 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.214758 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:43.226994 (+ 12236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:43.227001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:43.227006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:43.227065 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:43.231085 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.232322 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.232328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.232331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.232511 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.232978 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.232983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.232985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.233346 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:43.233717 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:43.234052 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:24:43.234349 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":313,"threads_started":1}
I20260927 15:24:43.235739 20005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:43.333002 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 19882
I20260927 15:24:43.362082 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20013
I20260927 15:24:43.384671 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:43.363333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.2.246.67:36545)
0927 15:24:43.364244 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:43.364252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:43.364283 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:24:43.364379 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:43.364402 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:43.364542 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:43.364732 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:43.367352 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.368959 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.371282 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:43.371850 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:43.372222 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:43.377030 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:43.377038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:43.377042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:43.377102 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:43.380674 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.381699 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.381712 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.381715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.382013 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:43.382521 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:43.382527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:43.382531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:43.382943 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:43.383309 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:43.383709 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:24:43.384187 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":824,"threads_started":1}
I20260927 15:24:43.386061 20137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:43.483955 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20013
2026-09-27T15:24:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66313 ms)
[----------] 30 tests from SecurityITest (332053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:24:43 dist-test-slave-14jq krb5kdc[20207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:24:43 dist-test-slave-14jq krb5kdc[20207](info): set up 2 sockets
Sep 27 15:24:43 dist-test-slave-14jq krb5kdc[20207](info): commencing operation
krb5kdc: starting...
W20260927 15:24:44.642230 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:24:44 dist-test-slave-14jq krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:24:44.941783 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46167
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:45.320305 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:45.320636 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:45.320793 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:45.329187 20220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:45.329329 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:45.329380 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:45.329519 20220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:45.329588 20220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:45.340062 20220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:45.341536 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:45.343045 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:45.356958 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:45.357012 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:45.357463 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:45.358971 20220 server_base.cc:1034] running on GCE node
I20260927 15:24:45.360029 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:45.361946 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:45.363308 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790522685363231 us; error 60 us; skew 500 ppm
I20260927 15:24:45.363806 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:45 dist-test-slave-14jq krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:45.368041 20220 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:24:45.370209 20220 webserver.cc:460] Webserver started at http://127.2.246.126:36321/ using document root <none> and password file <none>
I20260927 15:24:45.371073 20220 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:45.371240 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:45.371592 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:45.375021 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "e42556e38ef043309189bcb4c5e5b6ff"
format_stamp: "Formatted at 2026-09-27 15:24:45 on dist-test-slave-14jq"
server_key: "cb9954f427241cbc498f848c90216139"
server_key_iv: "f6e3d83fd9c49c3f4517476940027726"
server_key_version: "encryptionkey@0"
I20260927 15:24:45.375895 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "e42556e38ef043309189bcb4c5e5b6ff"
format_stamp: "Formatted at 2026-09-27 15:24:45 on dist-test-slave-14jq"
server_key: "cb9954f427241cbc498f848c90216139"
server_key_iv: "f6e3d83fd9c49c3f4517476940027726"
server_key_version: "encryptionkey@0"
I20260927 15:24:45.382230 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:24:45.386551 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:45.388698 20220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:24:45.388971 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "e42556e38ef043309189bcb4c5e5b6ff"
format_stamp: "Formatted at 2026-09-27 15:24:45 on dist-test-slave-14jq"
server_key: "cb9954f427241cbc498f848c90216139"
server_key_iv: "f6e3d83fd9c49c3f4517476940027726"
server_key_version: "encryptionkey@0"
I20260927 15:24:45.389205 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:45.405085 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:45.410413 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:45.410844 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:45.434791 20220 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:46167
I20260927 15:24:45.434906 20286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:46167 every 8 connection(s)
I20260927 15:24:45.436479 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:45.442032 20287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:45.445247 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20220
I20260927 15:24:45.445523 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:24:45.446355 3033 external_mini_cluster.cc:1428] Setting key e1b37ede0d0e369663a5aea6ba0b4b13
I20260927 15:24:45.461773 20287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: Bootstrap starting.
Sep 27 15:24:45 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:45.466751 20287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:45.468622 20287 log.cc:824] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:45.473451 20287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: No bootstrap required, opened a new log
I20260927 15:24:45.476825 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:45.448959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.2.246.126:46167)
0927 15:24:45.450344 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:45.450359 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:45.450409 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:24:45.452320 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:45.452350 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:45.452924 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:45.453434 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:45.455136 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:45.456725 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:45.458263 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:45.459005 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:45.459218 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:45.465462 (+ 6244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:45.465477 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:45.465505 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:45.465579 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:45.470415 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:45.471532 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:45.471544 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:45.471555 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:45.471829 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:45.472370 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:45.472375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:45.472379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:45.473265 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:45.473654 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:45.474454 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 15:24:45.475066 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":824,"threads_started":1}
I20260927 15:24:45.483857 20287 raft_consensus.cc:348] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } }
I20260927 15:24:45.484434 20287 raft_consensus.cc:374] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:45.484561 20287 raft_consensus.cc:729] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42556e38ef043309189bcb4c5e5b6ff, State: Initialized, Role: FOLLOWER
I20260927 15:24:45.485407 20287 consensus_queue.cc:260] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } }
I20260927 15:24:45.485715 20287 raft_consensus.cc:388] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:45.485946 20287 raft_consensus.cc:482] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:45.486155 20287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:45.489737 20287 raft_consensus.cc:504] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } }
I20260927 15:24:45.490653 20287 leader_election.cc:302] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e42556e38ef043309189bcb4c5e5b6ff; no voters:
I20260927 15:24:45.491746 20287 leader_election.cc:288] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:24:45.491993 20292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:45.493643 20292 raft_consensus.cc:686] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [term 1 LEADER]: Becoming Leader. State: Replica: e42556e38ef043309189bcb4c5e5b6ff, State: Running, Role: LEADER
I20260927 15:24:45.494505 20292 consensus_queue.cc:237] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } }
I20260927 15:24:45.494891 20287 sys_catalog.cc:564] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:45.498274 20294 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e42556e38ef043309189bcb4c5e5b6ff. Latest consensus state: current_term: 1 leader_uuid: "e42556e38ef043309189bcb4c5e5b6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } } }
I20260927 15:24:45.498817 20294 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:45.498386 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e42556e38ef043309189bcb4c5e5b6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42556e38ef043309189bcb4c5e5b6ff" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46167 } } }
I20260927 15:24:45.499061 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:45.502903 20300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:45.508872 20300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:45.520084 20300 catalog_manager.cc:1347] Generated new cluster ID: 457d19b19dcd46fe95e97b7723f2ac4e
I20260927 15:24:45.520278 20300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:45.545390 20300 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:24:45.546895 20300 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:45.556764 20300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: Generated new TSK 0
I20260927 15:24:45.558005 20300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:24:45.631408 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:45.985649 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:45.986030 20313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:45.986135 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:45.993701 20313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:45.993878 20313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:45.994064 20313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:46.005229 20313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:46.006788 20313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:46.008185 20313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:46.024716 20318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:46.024842 20321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:46.025761 20319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:46.026315 20313 server_base.cc:1034] running on GCE node
I20260927 15:24:46.026989 20313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:46.028066 20313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:46.029359 20313 hybrid_clock.cc:648] HybridClock initialized: now 1790522686029256 us; error 57 us; skew 500 ppm
I20260927 15:24:46.029770 20313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:46 dist-test-slave-14jq krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:46.033540 20313 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:24:46.035422 20313 webserver.cc:460] Webserver started at http://127.2.246.65:34193/ using document root <none> and password file <none>
I20260927 15:24:46.036166 20313 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:46.036327 20313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:46.036656 20313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:46.039805 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "218f6f5e72c74ad387e8c43a6c7b190c"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "33d4b25ffcf55e6e02d5c693f5e587c8"
server_key_iv: "71025a6fdf78bcfd6bcc3c1100458002"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.040640 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "218f6f5e72c74ad387e8c43a6c7b190c"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "33d4b25ffcf55e6e02d5c693f5e587c8"
server_key_iv: "71025a6fdf78bcfd6bcc3c1100458002"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.046722 20313 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:24:46.051158 20328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.053064 20313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:24:46.053330 20313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "218f6f5e72c74ad387e8c43a6c7b190c"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "33d4b25ffcf55e6e02d5c693f5e587c8"
server_key_iv: "71025a6fdf78bcfd6bcc3c1100458002"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.053580 20313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:46.079793 20313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:46.084785 20313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:46.085125 20313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:46.086308 20313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:46.087920 20313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:46.088029 20313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.088195 20313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:46.088272 20313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.128738 20313 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:33105
I20260927 15:24:46.129078 20440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:33105 every 8 connection(s)
I20260927 15:24:46.130620 20313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:46.139742 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20313
I20260927 15:24:46.140048 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:24:46.141023 3033 external_mini_cluster.cc:1428] Setting key 19fe9875d6df744428ffecb9dfcfade2
Sep 27 15:24:46 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:46.159484 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:46.134456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:35317 (local address 127.2.246.126:46167)
0927 15:24:46.135205 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:46.135214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:46.136715 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 15:24:46.139281 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:46.139291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:46.139434 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:46.139639 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:46.141750 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.142980 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.145203 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.146089 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.146319 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:46.150305 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:46.150317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:46.150327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:46.150402 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:46.154884 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:46.156151 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:46.156156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:46.156159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:46.156366 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:46.156989 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:46.156992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:46.156994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:46.157367 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:46.157500 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:46.158663 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0927 15:24:46.158975 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":379,"threads_started":1}
I20260927 15:24:46.160633 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:46.135355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46167 (local address 127.2.246.65:35317)
0927 15:24:46.136456 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:46.136525 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:46.138816 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:46.140024 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:46.140035 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:46.140740 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:46.141465 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:46.141485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.143269 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.143275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:46.144975 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:46.144991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.146391 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.146396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:46.146609 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:46.147403 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:46.147427 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:46.149915 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:46.155145 (+ 5230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:46.155155 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:46.155171 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:46.155949 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:46.156525 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:46.156534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:46.156539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:46.156842 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:46.157534 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:46.157545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:46.158098 (+ 553us) client_negotiation.cc:769] Sending connection context
0927 15:24:46.158434 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 15:24:46.159096 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":414,"threads_started":1}
I20260927 15:24:46.163118 20441 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46167
I20260927 15:24:46.163539 20441 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:46.164631 20441 heartbeater.cc:511] Master 127.2.246.126:46167 requested a full tablet report, sending...
I20260927 15:24:46.167738 20252 ts_manager.cc:194] Registered new tserver with Master: 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105)
I20260927 15:24:46.170225 20252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:35317
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:24:46.211730 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:46.563230 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:46.563535 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:46.563673 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:46.571367 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:46.571487 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:46.571664 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:46.586092 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:46.587422 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:46.588713 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:46.603430 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:46.603463 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:46.603459 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:46.605374 20447 server_base.cc:1034] running on GCE node
I20260927 15:24:46.606127 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:46.607200 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:46.608428 20447 hybrid_clock.cc:648] HybridClock initialized: now 1790522686608360 us; error 63 us; skew 500 ppm
I20260927 15:24:46.608801 20447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:46 dist-test-slave-14jq krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:46.612287 20447 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:24:46.614181 20447 webserver.cc:460] Webserver started at http://127.2.246.66:45943/ using document root <none> and password file <none>
I20260927 15:24:46.614953 20447 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:46.615105 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:46.615434 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:46.618553 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "87ac77d4029f440084550e99bfda6493"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "5beb7a1f3eff80ade25335fdae317cc8"
server_key_iv: "23056e814c97c21ef6dab143e78f01ac"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.619427 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "87ac77d4029f440084550e99bfda6493"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "5beb7a1f3eff80ade25335fdae317cc8"
server_key_iv: "23056e814c97c21ef6dab143e78f01ac"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.625653 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:24:46.630089 20462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.631783 20447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:24:46.632020 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "87ac77d4029f440084550e99bfda6493"
format_stamp: "Formatted at 2026-09-27 15:24:46 on dist-test-slave-14jq"
server_key: "5beb7a1f3eff80ade25335fdae317cc8"
server_key_iv: "23056e814c97c21ef6dab143e78f01ac"
server_key_version: "encryptionkey@0"
I20260927 15:24:46.632257 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:46.655279 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:46.660240 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:46.660549 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:46.661787 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:46.663290 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:46.663393 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.663546 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:46.663618 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:46.703472 20447 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:44293
I20260927 15:24:46.703593 20574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:44293 every 8 connection(s)
I20260927 15:24:46.704845 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:46.712711 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20447
I20260927 15:24:46.712936 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:24:46.713474 3033 external_mini_cluster.cc:1428] Setting key 71c1503514d5aa87c8791fd7841b56e2
Sep 27 15:24:46 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:46.734048 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:46.708264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:35159 (local address 127.2.246.126:46167)
0927 15:24:46.708921 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:46.708926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:46.709784 (+ 858us) server_negotiation.cc:407] Connection header received
0927 15:24:46.711779 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:46.711785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:46.711888 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:46.712044 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:46.713751 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.714755 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.718053 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.718532 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.718653 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:46.724792 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:46.724803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:46.724812 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:46.724881 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:46.729115 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:46.730507 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:46.730513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:46.730516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:46.730717 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:46.731374 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:46.731380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:46.731383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:46.731834 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:46.732009 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:46.733195 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0927 15:24:46.733473 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":384,"threads_started":1}
I20260927 15:24:46.735333 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:46.708678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46167 (local address 127.2.246.66:35159)
0927 15:24:46.709538 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:46.709584 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:46.711461 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:46.712255 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:46.712264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:46.712791 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:46.713480 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:46.713499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.715014 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.715019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:46.717777 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:46.717810 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:46.718905 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:46.718919 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:46.719379 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:46.720493 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:46.720804 (+ 311us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:46.724394 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:46.729461 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:46.729477 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:46.729503 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:46.730294 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:46.730853 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:46.730861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:46.730867 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:46.731138 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:46.732035 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:46.732048 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:46.732597 (+ 549us) client_negotiation.cc:769] Sending connection context
0927 15:24:46.732965 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 15:24:46.733658 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":366,"threads_started":1}
I20260927 15:24:46.737501 20575 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46167
I20260927 15:24:46.738056 20575 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:46.739108 20575 heartbeater.cc:511] Master 127.2.246.126:46167 requested a full tablet report, sending...
I20260927 15:24:46.741230 20252 ts_manager.cc:194] Registered new tserver with Master: 87ac77d4029f440084550e99bfda6493 (127.2.246.66:44293)
I20260927 15:24:46.742378 20252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:35159
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:24:46.786998 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:47.142958 20581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:47.143318 20581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:47.143487 20581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:47.151369 20581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:47.151538 20581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:47.151727 20581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:47.162564 20581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:47.164310 20581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:47.165982 20581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:24:47.175513 20441 heartbeater.cc:503] Master 127.2.246.126:46167 was elected leader, sending a full tablet report...
W20260927 15:24:47.184512 20589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:47.184437 20587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:47.184563 20586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:47.185439 20581 server_base.cc:1034] running on GCE node
I20260927 15:24:47.186347 20581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:47.187489 20581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:47.188784 20581 hybrid_clock.cc:648] HybridClock initialized: now 1790522687188704 us; error 70 us; skew 500 ppm
I20260927 15:24:47.189198 20581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:47 dist-test-slave-14jq krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:47.193343 20581 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:24:47.195924 20581 webserver.cc:460] Webserver started at http://127.2.246.67:33603/ using document root <none> and password file <none>
I20260927 15:24:47.196907 20581 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:47.197044 20581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:47.197348 20581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:47.200843 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "4183dc05d0e94f02ad6c0bca6d9d152c"
format_stamp: "Formatted at 2026-09-27 15:24:47 on dist-test-slave-14jq"
server_key: "e874c23201c0b57e689bd17abe2a6696"
server_key_iv: "6b24660490501f7ce31fe53223fa01c7"
server_key_version: "encryptionkey@0"
I20260927 15:24:47.201776 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "4183dc05d0e94f02ad6c0bca6d9d152c"
format_stamp: "Formatted at 2026-09-27 15:24:47 on dist-test-slave-14jq"
server_key: "e874c23201c0b57e689bd17abe2a6696"
server_key_iv: "6b24660490501f7ce31fe53223fa01c7"
server_key_version: "encryptionkey@0"
I20260927 15:24:47.208683 20581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:24:47.215200 20596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:47.217597 20581 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:24:47.217965 20581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "4183dc05d0e94f02ad6c0bca6d9d152c"
format_stamp: "Formatted at 2026-09-27 15:24:47 on dist-test-slave-14jq"
server_key: "e874c23201c0b57e689bd17abe2a6696"
server_key_iv: "6b24660490501f7ce31fe53223fa01c7"
server_key_version: "encryptionkey@0"
I20260927 15:24:47.218248 20581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:47.234614 20581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:47.240187 20581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:47.240520 20581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:47.241622 20581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:47.243209 20581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:47.243317 20581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:47.243476 20581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:47.243554 20581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:47.286259 20581 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:35199
I20260927 15:24:47.286298 20708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:35199 every 8 connection(s)
I20260927 15:24:47.287614 20581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:47.292220 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20581
I20260927 15:24:47.292426 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:24:47.292929 3033 external_mini_cluster.cc:1428] Setting key c25ee8182bea9f5442b1fb5094004cbc
Sep 27 15:24:47 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:47.316766 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.291452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:51353 (local address 127.2.246.126:46167)
0927 15:24:47.292179 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.292187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.293918 (+ 1731us) server_negotiation.cc:407] Connection header received
0927 15:24:47.297934 (+ 4016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.297938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.298083 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.298250 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:47.300968 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.302013 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.304545 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.304991 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.305116 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.310278 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:47.310284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:47.310287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:47.310328 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:47.313255 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.314210 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.314218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.314224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.314355 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.314830 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.314835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.314840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.315140 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:47.315324 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.316076 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.316327 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":347,"threads_started":1}
I20260927 15:24:47.317942 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.292297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:46167 (local address 127.2.246.67:51353)
0927 15:24:47.293620 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.293687 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.297352 (+ 3665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.298756 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.298770 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.299758 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:47.300629 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.300660 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.302366 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.302376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.304272 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.304300 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.305336 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.305343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.305689 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.306676 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:47.306709 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:47.309967 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:47.313457 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:47.313469 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:47.313495 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:47.314084 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:47.314476 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:47.314481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:47.314485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:47.314706 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:47.315276 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:47.315284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:47.315686 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.315960 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.316569 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":440,"threads_started":1}
I20260927 15:24:47.320317 20709 heartbeater.cc:347] Connected to a master server at 127.2.246.126:46167
I20260927 15:24:47.320755 20709 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:47.321668 20709 heartbeater.cc:511] Master 127.2.246.126:46167 requested a full tablet report, sending...
I20260927 15:24:47.324232 20252 ts_manager.cc:194] Registered new tserver with Master: 4183dc05d0e94f02ad6c0bca6d9d152c (127.2.246.67:35199)
I20260927 15:24:47.325582 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:24:47.325700 20252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:51353
I20260927 15:24:47.349941 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.331759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.2.246.126:46167)
0927 15:24:47.332073 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.332081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.332634 (+ 553us) server_negotiation.cc:407] Connection header received
0927 15:24:47.332865 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.332869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.332964 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.333084 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:47.335406 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.336823 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.338908 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.339364 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.339515 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.343147 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:47.343153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:47.343156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:47.343236 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:47.346254 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.347231 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.347236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.347238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.347388 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.348101 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.348105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.348108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.348528 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:47.348752 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.349294 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.349515 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 15:24:47.355993 20252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:24:47.363559 20252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:24:47.394325 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.380536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.2.246.67:35199)
0927 15:24:47.382839 (+ 2303us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.382853 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.382903 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:24:47.383043 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.383055 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.383425 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.383969 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.385345 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.387164 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.390749 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.392506 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.392929 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.393238 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.393557 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.393815 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2061,"thread_start_us":1871,"threads_started":1}
I20260927 15:24:47.394541 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.380097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35199 (local address 127.0.0.1:46888)
0927 15:24:47.381398 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.381472 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.381866 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.383759 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.383768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.384367 (+ 599us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.384923 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.384942 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.387351 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.387358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.390505 (+ 3147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.390521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.392816 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.392824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.393202 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.393276 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.393455 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.393819 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":67,"spinlock_wait_cycles":1232768,"thread_start_us":438,"threads_started":1}
I20260927 15:24:47.395303 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.382106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44293 (local address 127.0.0.1:53394)
0927 15:24:47.385158 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.385191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.385304 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.387389 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.387395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.387447 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.388129 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.388146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.389978 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.389984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.392149 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.392167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.394321 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.394328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.394567 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.394605 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.394680 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.394839 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2905,"mutex_wait_us":3,"thread_start_us":805,"threads_started":1}
I20260927 15:24:47.396413 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.381794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.2.246.66:44293)
0927 15:24:47.385938 (+ 4144us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.385949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.385988 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:24:47.386107 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.386117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.386456 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.386705 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.388440 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.389775 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.392406 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.394036 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.395605 (+ 1569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.395822 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.395896 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.396085 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":3741,"threads_started":1}
I20260927 15:24:47.398041 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.380838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33105 (local address 127.0.0.1:35622)
0927 15:24:47.383269 (+ 2431us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.383291 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.385158 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.388432 (+ 3274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.388437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.388466 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.388960 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.388973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.391585 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.391592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.394019 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.394039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.396986 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.396994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.397218 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.397257 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.397375 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.397567 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2273,"mutex_wait_us":1,"thread_start_us":3995,"threads_started":1}
I20260927 15:24:47.399164 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.381570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.2.246.65:33105)
0927 15:24:47.387491 (+ 5921us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.387503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.387539 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:24:47.387666 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.387674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.387931 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.388155 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.389427 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.391331 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.394291 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.396207 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.396373 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.396684 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.398534 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.398799 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5696,"thread_start_us":5495,"threads_started":1}
I20260927 15:24:47.399385 20644 tablet_service.cc:1467] Processing CreateTablet for tablet 3105cd12374043d888a56362249e1921 (DEFAULT_TABLE table=test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:47.400475 20510 tablet_service.cc:1467] Processing CreateTablet for tablet 3105cd12374043d888a56362249e1921 (DEFAULT_TABLE table=test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:47.401152 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3105cd12374043d888a56362249e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:47.402046 20510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3105cd12374043d888a56362249e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:47.404225 20376 tablet_service.cc:1467] Processing CreateTablet for tablet 3105cd12374043d888a56362249e1921 (DEFAULT_TABLE table=test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:47.405930 20376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3105cd12374043d888a56362249e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:47.419716 20728 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Bootstrap starting.
I20260927 15:24:47.423208 20730 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Bootstrap starting.
I20260927 15:24:47.425101 20729 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Bootstrap starting.
I20260927 15:24:47.425118 20728 tablet_bootstrap.cc:654] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:47.426982 20728 log.cc:824] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:47.427553 20730 tablet_bootstrap.cc:654] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:47.428834 20729 tablet_bootstrap.cc:654] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:47.429018 20730 log.cc:824] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:47.430133 20729 log.cc:824] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:47.432971 20730 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: No bootstrap required, opened a new log
I20260927 15:24:47.433290 20730 ts_tablet_manager.cc:1397] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260927 15:24:47.433386 20728 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: No bootstrap required, opened a new log
I20260927 15:24:47.433547 20729 tablet_bootstrap.cc:492] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: No bootstrap required, opened a new log
I20260927 15:24:47.433810 20729 ts_tablet_manager.cc:1397] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 15:24:47.433791 20728 ts_tablet_manager.cc:1397] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260927 15:24:47.444849 20730 raft_consensus.cc:348] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.444849 20729 raft_consensus.cc:348] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.445356 20729 raft_consensus.cc:374] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:47.445356 20730 raft_consensus.cc:374] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:47.445457 20730 raft_consensus.cc:729] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218f6f5e72c74ad387e8c43a6c7b190c, State: Initialized, Role: FOLLOWER
I20260927 15:24:47.445452 20729 raft_consensus.cc:729] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ac77d4029f440084550e99bfda6493, State: Initialized, Role: FOLLOWER
I20260927 15:24:47.446332 20729 consensus_queue.cc:260] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.446332 20730 consensus_queue.cc:260] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.449467 20728 raft_consensus.cc:348] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.450145 20728 raft_consensus.cc:374] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:47.450278 20728 raft_consensus.cc:729] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4183dc05d0e94f02ad6c0bca6d9d152c, State: Initialized, Role: FOLLOWER
I20260927 15:24:47.452960 20728 consensus_queue.cc:260] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.454073 20575 heartbeater.cc:503] Master 127.2.246.126:46167 was elected leader, sending a full tablet report...
I20260927 15:24:47.455647 20729 ts_tablet_manager.cc:1428] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Time spent starting tablet: real 0.022s user 0.009s sys 0.010s
I20260927 15:24:47.457461 20730 ts_tablet_manager.cc:1428] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Time spent starting tablet: real 0.024s user 0.023s sys 0.002s
W20260927 15:24:47.458428 20576 tablet.cc:2369] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:47.460713 20709 heartbeater.cc:503] Master 127.2.246.126:46167 was elected leader, sending a full tablet report...
I20260927 15:24:47.461548 20728 ts_tablet_manager.cc:1428] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Time spent starting tablet: real 0.028s user 0.014s sys 0.008s
W20260927 15:24:47.539898 20710 tablet.cc:2369] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:24:47.634609 20442 tablet.cc:2369] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:47.721222 20734 raft_consensus.cc:482] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:47.721604 20734 raft_consensus.cc:504] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.722128 20736 raft_consensus.cc:482] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:47.722505 20736 raft_consensus.cc:504] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.723870 20734 leader_election.cc:288] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4183dc05d0e94f02ad6c0bca6d9d152c (127.2.246.67:35199), 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105)
I20260927 15:24:47.724712 20736 leader_election.cc:288] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ac77d4029f440084550e99bfda6493 (127.2.246.66:44293), 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105)
I20260927 15:24:47.736738 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.724339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35199 (local address 127.2.246.66:50481)
0927 15:24:47.725572 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.725592 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.725885 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.726762 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.726768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.726809 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.727339 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.727358 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.729538 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.729547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.732657 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.732686 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.735568 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.735577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.735851 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.735893 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.736165 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.736340 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":959,"threads_started":1}
I20260927 15:24:47.736707 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.725062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:50481 (local address 127.2.246.67:35199)
0927 15:24:47.725613 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.725619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.725718 (+ 99us) server_negotiation.cc:407] Connection header received
0927 15:24:47.726180 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.726192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.726366 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.726457 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.727556 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.729223 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.732968 (+ 3745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.734986 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.735184 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.735291 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.736088 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.736331 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":434}
I20260927 15:24:47.737989 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.725120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:44293 (local address 127.2.246.67:53691)
0927 15:24:47.725467 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.725493 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.725645 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.726649 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.726657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.726698 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.727015 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.727032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.729654 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.729664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.732735 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.732765 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.735577 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.735588 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.735889 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.735949 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.736088 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.737544 (+ 1456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":6}
I20260927 15:24:47.738533 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.725219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:53691 (local address 127.2.246.66:44293)
0927 15:24:47.726041 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.726050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.726077 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:24:47.726176 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.726186 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.726291 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.726404 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.727442 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.729312 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.733100 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.735005 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.735297 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.735437 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.737630 (+ 2193us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.737800 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577}
I20260927 15:24:47.739945 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ac77d4029f440084550e99bfda6493" is_pre_election: true
I20260927 15:24:47.740113 20664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "87ac77d4029f440084550e99bfda6493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" is_pre_election: true
I20260927 15:24:47.740473 20530 raft_consensus.cc:2454] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4183dc05d0e94f02ad6c0bca6d9d152c in term 0.
I20260927 15:24:47.740722 20664 raft_consensus.cc:2454] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ac77d4029f440084550e99bfda6493 in term 0.
I20260927 15:24:47.741328 20600 leader_election.cc:302] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4183dc05d0e94f02ad6c0bca6d9d152c, 87ac77d4029f440084550e99bfda6493; no voters:
I20260927 15:24:47.741456 20466 leader_election.cc:302] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4183dc05d0e94f02ad6c0bca6d9d152c, 87ac77d4029f440084550e99bfda6493; no voters:
I20260927 15:24:47.742040 20734 raft_consensus.cc:2781] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:47.742115 20736 raft_consensus.cc:2781] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:47.742218 20734 raft_consensus.cc:482] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:47.742332 20734 raft_consensus.cc:3037] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:47.742326 20736 raft_consensus.cc:482] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:47.742458 20736 raft_consensus.cc:3037] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:47.745222 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.724630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:46641 (local address 127.2.246.65:33105)
0927 15:24:47.724988 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.724997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.736104 (+ 11107us) server_negotiation.cc:407] Connection header received
0927 15:24:47.736548 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.736555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.736642 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.736732 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.737672 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.738871 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.742637 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.743892 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.744048 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.744134 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.744770 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.744931 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260927 15:24:47.745702 20734 raft_consensus.cc:504] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.746318 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.724353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33105 (local address 127.2.246.66:46641)
0927 15:24:47.735983 (+ 11630us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.736008 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.736342 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.737036 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.737040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.737063 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.737488 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.737501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.739444 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.739451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.742218 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.742236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.744360 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.744369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.744559 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.744603 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.745672 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.745829 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10983,"mutex_wait_us":1,"thread_start_us":12594,"threads_started":1}
I20260927 15:24:47.746241 20736 raft_consensus.cc:504] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.746869 20734 leader_election.cc:288] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [CANDIDATE]: Term 1 election: Requested vote from peers 4183dc05d0e94f02ad6c0bca6d9d152c (127.2.246.67:35199), 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105)
I20260927 15:24:47.747573 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.725730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:36791 (local address 127.2.246.65:33105)
0927 15:24:47.736562 (+ 10832us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.736568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.739003 (+ 2435us) server_negotiation.cc:407] Connection header received
0927 15:24:47.739114 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.739119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.739205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.739808 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:47.740169 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.741504 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.744209 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.745646 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.746797 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.746894 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.746994 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.747167 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10643,"thread_start_us":10377,"threads_started":1}
I20260927 15:24:47.747785 20736 leader_election.cc:288] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [CANDIDATE]: Term 1 election: Requested vote from peers 87ac77d4029f440084550e99bfda6493 (127.2.246.66:44293), 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105)
I20260927 15:24:47.747799 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "87ac77d4029f440084550e99bfda6493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" is_pre_election: true
I20260927 15:24:47.747903 20395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "87ac77d4029f440084550e99bfda6493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218f6f5e72c74ad387e8c43a6c7b190c"
I20260927 15:24:47.748407 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.725352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:33105 (local address 127.2.246.67:36791)
0927 15:24:47.738635 (+ 13283us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:47.738663 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:47.738839 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:47.739364 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:47.739369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:47.739396 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:47.739649 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.739661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.741661 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.741668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.743965 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:47.743982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.745903 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.745914 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:47.746146 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.746196 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 15:24:47.746284 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:24:47.746441 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13131,"thread_start_us":13698,"threads_started":1}
I20260927 15:24:47.748485 20396 raft_consensus.cc:2454] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ac77d4029f440084550e99bfda6493 in term 0.
I20260927 15:24:47.748549 20664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "87ac77d4029f440084550e99bfda6493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c"
I20260927 15:24:47.748893 20664 raft_consensus.cc:2379] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87ac77d4029f440084550e99bfda6493 in current term 1: Already voted for candidate 4183dc05d0e94f02ad6c0bca6d9d152c in this term.
I20260927 15:24:47.749449 20394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218f6f5e72c74ad387e8c43a6c7b190c"
I20260927 15:24:47.749588 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ac77d4029f440084550e99bfda6493"
I20260927 15:24:47.749645 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3105cd12374043d888a56362249e1921" candidate_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" is_pre_election: true
I20260927 15:24:47.749918 20530 raft_consensus.cc:2379] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4183dc05d0e94f02ad6c0bca6d9d152c in current term 1: Already voted for candidate 87ac77d4029f440084550e99bfda6493 in this term.
I20260927 15:24:47.749861 20394 raft_consensus.cc:3037] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:47.750554 20466 leader_election.cc:302] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ac77d4029f440084550e99bfda6493; no voters: 218f6f5e72c74ad387e8c43a6c7b190c, 4183dc05d0e94f02ad6c0bca6d9d152c
I20260927 15:24:47.751114 20734 raft_consensus.cc:2726] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 15:24:47.753113 20394 raft_consensus.cc:2454] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4183dc05d0e94f02ad6c0bca6d9d152c in term 1.
I20260927 15:24:47.753679 20598 leader_election.cc:302] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 218f6f5e72c74ad387e8c43a6c7b190c, 4183dc05d0e94f02ad6c0bca6d9d152c; no voters: 87ac77d4029f440084550e99bfda6493
I20260927 15:24:47.754163 20736 raft_consensus.cc:2781] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:47.754724 20736 raft_consensus.cc:686] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 LEADER]: Becoming Leader. State: Replica: 4183dc05d0e94f02ad6c0bca6d9d152c, State: Running, Role: LEADER
I20260927 15:24:47.755318 20736 consensus_queue.cc:237] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } }
I20260927 15:24:47.761116 20252 catalog_manager.cc:5495] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4183dc05d0e94f02ad6c0bca6d9d152c (127.2.246.67). New cstate: current_term: 1 leader_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183dc05d0e94f02ad6c0bca6d9d152c" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 } health_report { overall_health: UNKNOWN } } }
I20260927 15:24:47.865751 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.858374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.2.246.67:35199)
0927 15:24:47.858707 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.858713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.859012 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:24:47.859256 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.859261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.859411 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.859590 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:47.860150 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.860998 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.862898 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.863468 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.863586 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.864290 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:47.864909 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:24:47.865030 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.865244 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.865324 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:24:47.876097 20530 raft_consensus.cc:1261] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Refusing update from remote peer 4183dc05d0e94f02ad6c0bca6d9d152c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:47.876147 20394 raft_consensus.cc:1261] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 1 FOLLOWER]: Refusing update from remote peer 4183dc05d0e94f02ad6c0bca6d9d152c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:47.877300 20736 consensus_queue.cc:1035] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ac77d4029f440084550e99bfda6493" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:47.877802 20741 consensus_queue.cc:1035] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [LEADER]: Connected to new peer: Peer: permanent_uuid: "218f6f5e72c74ad387e8c43a6c7b190c" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:47.894392 20745 mvcc.cc:205] Tried to move back new op lower bound from 7333980929527926784 to 7333980929050435584. Current Snapshot: MvccSnapshot[applied={T|T < 7333980929527926784}]
I20260927 15:24:47.896126 20746 mvcc.cc:205] Tried to move back new op lower bound from 7333980929527926784 to 7333980929050435584. Current Snapshot: MvccSnapshot[applied={T|T < 7333980929527926784}]
I20260927 15:24:47.907482 20252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:47.907882 20252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:47.912348 20252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42556e38ef043309189bcb4c5e5b6ff: Sending DeleteTablet for 3 replicas of tablet 3105cd12374043d888a56362249e1921
I20260927 15:24:47.913941 20510 tablet_service.cc:1514] Processing DeleteTablet for tablet 3105cd12374043d888a56362249e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:47 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260927 15:24:47.913949 20644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3105cd12374043d888a56362249e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:47 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:46888
I20260927 15:24:47.914181 20376 tablet_service.cc:1514] Processing DeleteTablet for tablet 3105cd12374043d888a56362249e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:47 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:35622
I20260927 15:24:47.916771 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20313
I20260927 15:24:47.926919 20757 tablet_replica.cc:330] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: stopping tablet replica
I20260927 15:24:47.927737 20757 raft_consensus.cc:2229] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:47.928540 20756 tablet_replica.cc:330] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: stopping tablet replica
I20260927 15:24:47.928797 20757 raft_consensus.cc:2258] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:47.929132 20756 raft_consensus.cc:2229] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:47.929693 20756 raft_consensus.cc:2258] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:47.931474 20755 tablet_replica.cc:330] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: stopping tablet replica
I20260927 15:24:47.931998 20755 raft_consensus.cc:2229] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:24:47.932149 20757 ts_tablet_manager.cc:1905] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:47.932685 20756 ts_tablet_manager.cc:1905] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:47.932684 20755 raft_consensus.cc:2258] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:47.938829 20755 ts_tablet_manager.cc:1905] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:47.941167 20756 ts_tablet_manager.cc:1918] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:47.941161 20757 ts_tablet_manager.cc:1918] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:47.941411 20757 log.cc:1196] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/3105cd12374043d888a56362249e1921
I20260927 15:24:47.941411 20756 log.cc:1196] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/3105cd12374043d888a56362249e1921
I20260927 15:24:47.942087 20757 ts_tablet_manager.cc:1939] T 3105cd12374043d888a56362249e1921 P 218f6f5e72c74ad387e8c43a6c7b190c: Deleting consensus metadata
I20260927 15:24:47.942090 20756 ts_tablet_manager.cc:1939] T 3105cd12374043d888a56362249e1921 P 87ac77d4029f440084550e99bfda6493: Deleting consensus metadata
I20260927 15:24:47.944749 20237 catalog_manager.cc:4841] TS 218f6f5e72c74ad387e8c43a6c7b190c (127.2.246.65:33105): tablet 3105cd12374043d888a56362249e1921 (table test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]) successfully deleted
I20260927 15:24:47.944790 20239 catalog_manager.cc:4841] TS 87ac77d4029f440084550e99bfda6493 (127.2.246.66:44293): tablet 3105cd12374043d888a56362249e1921 (table test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]) successfully deleted
Sep 27 15:24:47 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:24:47.951805 20755 ts_tablet_manager.cc:1918] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:47.952067 20755 log.cc:1196] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/3105cd12374043d888a56362249e1921
I20260927 15:24:47.952646 20755 ts_tablet_manager.cc:1939] T 3105cd12374043d888a56362249e1921 P 4183dc05d0e94f02ad6c0bca6d9d152c: Deleting consensus metadata
I20260927 15:24:47.955437 20239 catalog_manager.cc:4841] TS 4183dc05d0e94f02ad6c0bca6d9d152c (127.2.246.67:35199): tablet 3105cd12374043d888a56362249e1921 (table test-table [id=3ca32a7725e94be3b703f2d15bbeb9a3]) successfully deleted
I20260927 15:24:47.957365 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:47.919075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.2.246.65:33105)
0927 15:24:47.919376 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:47.919385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:47.927316 (+ 7931us) server_negotiation.cc:407] Connection header received
0927 15:24:47.927576 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:47.927585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:47.927720 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:47.927888 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:47.932619 (+ 4731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.933816 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.939120 (+ 5304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:47.939843 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:47.939962 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:47.948425 (+ 8463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:47.948449 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:47.948472 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:47.948541 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:47.952799 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.954032 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.954044 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.954050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.954245 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:47.954926 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:47.954932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:47.954935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:47.955493 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:47.955803 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:47.956576 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 15:24:47.956932 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:24:47.958376 20436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:48.065461 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20313
I20260927 15:24:48.091547 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20447
Sep 27 15:24:48 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:24:48.117184 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:48.092758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.2.246.66:44293)
0927 15:24:48.092989 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:48.092995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:48.093021 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:24:48.093254 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:48.093292 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:48.093474 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:48.093589 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:48.095642 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:48.096617 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:48.098963 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:48.099473 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:48.099606 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:48.109749 (+ 10143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:48.109773 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:48.109796 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:48.109932 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:48.113669 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:48.114712 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:48.114720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:48.114723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:48.114876 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:48.115296 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:48.115300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:48.115302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:48.115763 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:48.116106 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:48.116465 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 15:24:48.116760 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:24:48.118036 20570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:48.244904 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20447
I20260927 15:24:48.273224 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20581
Sep 27 15:24:48 dist-test-slave-14jq krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:24:48.300912 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:48.274351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.2.246.67:35199)
0927 15:24:48.275285 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:48.275296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:48.275322 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:24:48.275412 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:48.275420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:48.275555 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:48.275690 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:48.278079 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:48.279603 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:48.282183 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:48.283038 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:48.283312 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:48.290746 (+ 7434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:48.290773 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:48.290796 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:48.290869 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:48.296480 (+ 5611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:48.297894 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:48.297907 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:48.297911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:48.298131 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:48.298699 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:48.298705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:48.298709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:48.299336 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:48.299734 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:48.300128 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 15:24:48.300440 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260927 15:24:48.301955 20704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:48.433202 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20581
I20260927 15:24:48.458439 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20220
I20260927 15:24:48.459973 20282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:48.596498 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20220
2026-09-27T15:24:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5121 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:24:48Z Disabled control of system clock
I20260927 15:24:48.698820 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:38817
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:24:49.096685 20766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:49.097023 20766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:49.097106 20766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:49.104832 20766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:49.104967 20766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:49.105007 20766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:49.105058 20766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:49.105098 20766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:49.114753 20766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:38817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:49.116065 20766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:49.117825 20766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:49.130995 20771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:49.131049 20772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:49.131108 20774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:49.133047 20766 server_base.cc:1034] running on GCE node
I20260927 15:24:49.134049 20766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:49.135710 20766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:49.136950 20766 hybrid_clock.cc:648] HybridClock initialized: now 1790522689136867 us; error 78 us; skew 500 ppm
I20260927 15:24:49.137395 20766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:49.140372 20766 webserver.cc:460] Webserver started at http://127.2.246.126:44791/ using document root <none> and password file <none>
I20260927 15:24:49.141203 20766 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:49.141299 20766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:49.141693 20766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:49.144824 20766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "599ef66385da4359b2181d629b14c638"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a633096e2f94bd96c63ebf796bce5b3b"
server_key_iv: "7ecb4b001eebfd207d6ff9efef9ed5bc"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.145668 20766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "599ef66385da4359b2181d629b14c638"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a633096e2f94bd96c63ebf796bce5b3b"
server_key_iv: "7ecb4b001eebfd207d6ff9efef9ed5bc"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.151794 20766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260927 15:24:49.156364 20780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:49.158301 20766 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:24:49.158560 20766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "599ef66385da4359b2181d629b14c638"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a633096e2f94bd96c63ebf796bce5b3b"
server_key_iv: "7ecb4b001eebfd207d6ff9efef9ed5bc"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.158952 20766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:49.183486 20766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:49.184366 20766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:49.184643 20766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:49.207674 20766 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:38817
I20260927 15:24:49.207800 20831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:38817 every 8 connection(s)
I20260927 15:24:49.209239 20766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:49.211896 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20766
I20260927 15:24:49.212059 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:24:49.212571 3033 external_mini_cluster.cc:1428] Setting key 8c19234405be97bcec14955341e47111
I20260927 15:24:49.217020 20832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:49.229574 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:49.214843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.2.246.126:38817)
0927 15:24:49.216250 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:49.216272 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:49.216342 (+ 70us) server_negotiation.cc:407] Connection header received
0927 15:24:49.218833 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:49.218870 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:49.219579 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:49.220093 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:49.220842 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.222599 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.224229 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.225040 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.225272 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:49.226830 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:49.226841 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:49.226859 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:49.227522 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:49.227612 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:49.227645 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:49.227835 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:49.228067 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:24:49.228433 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":435,"threads_started":1}
I20260927 15:24:49.232160 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: Bootstrap starting.
I20260927 15:24:49.236500 20832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:49.238065 20832 log.cc:824] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:49.242146 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: No bootstrap required, opened a new log
I20260927 15:24:49.252344 20832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } }
I20260927 15:24:49.253077 20832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:49.253232 20832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599ef66385da4359b2181d629b14c638, State: Initialized, Role: FOLLOWER
I20260927 15:24:49.254055 20832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } }
I20260927 15:24:49.254444 20832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:49.254632 20832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:49.254863 20832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:49.257495 20832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } }
I20260927 15:24:49.258283 20832 leader_election.cc:302] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599ef66385da4359b2181d629b14c638; no voters:
I20260927 15:24:49.259078 20832 leader_election.cc:288] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:24:49.259418 20837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:49.260432 20837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [term 1 LEADER]: Becoming Leader. State: Replica: 599ef66385da4359b2181d629b14c638, State: Running, Role: LEADER
I20260927 15:24:49.261104 20837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } }
I20260927 15:24:49.261987 20832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:49.265913 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599ef66385da4359b2181d629b14c638. Latest consensus state: current_term: 1 leader_uuid: "599ef66385da4359b2181d629b14c638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } } }
I20260927 15:24:49.266539 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:49.267602 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599ef66385da4359b2181d629b14c638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599ef66385da4359b2181d629b14c638" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38817 } } }
I20260927 15:24:49.267949 20843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:49.268121 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638 [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:49.277544 20843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:49.289587 20843 catalog_manager.cc:1347] Generated new cluster ID: a77a01cfa87f41f7bf4c48210559f59f
I20260927 15:24:49.289785 20843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:49.314453 20843 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:24:49.315963 20843 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:49.335415 20843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: Generated new TSK 0
I20260927 15:24:49.336568 20843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:24:49.357728 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--builtin_ntp_servers=127.2.246.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:24:49.824307 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:49.824702 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:49.824877 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:49.833328 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:49.833483 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:49.833716 20856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:49.845710 20856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:49.847616 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:49.850085 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:49.867821 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:49.867743 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:49.867697 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:49.872378 20856 server_base.cc:1034] running on GCE node
I20260927 15:24:49.873399 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:49.874905 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:49.876269 20856 hybrid_clock.cc:648] HybridClock initialized: now 1790522689876207 us; error 64 us; skew 500 ppm
I20260927 15:24:49.876713 20856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:49.880115 20856 webserver.cc:460] Webserver started at http://127.2.246.65:45835/ using document root <none> and password file <none>
I20260927 15:24:49.880993 20856 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:49.881141 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:49.881521 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:49.884883 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "3951c406c95d45c582bba127513d27c3"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a295c48268e84e7bd21dd25cd0f5129c"
server_key_iv: "7b37e4832b9490ccf36be5efa30d704a"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.885908 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "3951c406c95d45c582bba127513d27c3"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a295c48268e84e7bd21dd25cd0f5129c"
server_key_iv: "7b37e4832b9490ccf36be5efa30d704a"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.892215 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 15:24:49.897117 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:49.899281 20856 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260927 15:24:49.899518 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "3951c406c95d45c582bba127513d27c3"
format_stamp: "Formatted at 2026-09-27 15:24:49 on dist-test-slave-14jq"
server_key: "a295c48268e84e7bd21dd25cd0f5129c"
server_key_iv: "7b37e4832b9490ccf36be5efa30d704a"
server_key_version: "encryptionkey@0"
I20260927 15:24:49.899811 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:49.918236 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:49.919487 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:49.919924 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:49.921324 20856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:49.923511 20856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:49.923667 20856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:49.923863 20856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:49.923969 20856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:49.971060 20856 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:42991
I20260927 15:24:49.971179 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:42991 every 8 connection(s)
I20260927 15:24:49.973449 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:49.977550 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20856
I20260927 15:24:49.977770 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:24:49.978473 3033 external_mini_cluster.cc:1428] Setting key 88bfeea842c26451f837f876fadf38b6
I20260927 15:24:49.982275 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--builtin_ntp_servers=127.2.246.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:24:49.989120 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:49.977123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:36955 (local address 127.2.246.126:38817)
0927 15:24:49.977648 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:49.977677 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:49.979999 (+ 2322us) server_negotiation.cc:407] Connection header received
0927 15:24:49.982266 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:49.982272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:49.982385 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:49.982511 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:49.983749 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.984504 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.986231 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.986639 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.986780 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:49.988228 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:49.988233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:49.988236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:49.988257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:49.988311 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:49.988325 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:49.988425 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:49.988652 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:24:49.988759 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":303,"threads_started":1}
I20260927 15:24:49.990267 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:49.978188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38817 (local address 127.2.246.65:36955)
0927 15:24:49.979770 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:49.979852 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:49.981956 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:49.982825 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:49.982835 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:49.982936 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:49.983560 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:49.983577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.984686 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.984690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:49.986078 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:49.986091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:49.986886 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:49.986893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:49.987123 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:49.987888 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:49.987890 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:49.987930 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:49.988013 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:49.988424 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:49.988437 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:49.988447 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 15:24:49.988642 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 15:24:49.989077 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":410,"threads_started":1}
I20260927 15:24:49.992748 20983 heartbeater.cc:347] Connected to a master server at 127.2.246.126:38817
I20260927 15:24:49.993134 20983 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:49.994177 20983 heartbeater.cc:511] Master 127.2.246.126:38817 requested a full tablet report, sending...
I20260927 15:24:49.997198 20797 ts_manager.cc:194] Registered new tserver with Master: 3951c406c95d45c582bba127513d27c3 (127.2.246.65:42991)
I20260927 15:24:49.999096 20797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.65:36955
W20260927 15:24:50.361001 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:50.361382 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:50.361567 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:50.369423 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:50.369596 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:50.369776 20987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:50.379644 20987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:50.381330 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:50.383351 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:50.400722 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:50.400691 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:50.400624 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:50.401736 20987 server_base.cc:1034] running on GCE node
I20260927 15:24:50.402632 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:50.404119 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:50.405514 20987 hybrid_clock.cc:648] HybridClock initialized: now 1790522690405415 us; error 95 us; skew 500 ppm
I20260927 15:24:50.406070 20987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:50.409687 20987 webserver.cc:460] Webserver started at http://127.2.246.66:37679/ using document root <none> and password file <none>
I20260927 15:24:50.410880 20987 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:50.411067 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:50.411558 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:50.416239 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "50f4a63409004abea7b3f26310cab611"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "d8d2a52861a89bbc48382ea1aad5b127"
server_key_iv: "44a2929a980ac0a4eba266e4f4d32c50"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.417361 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "50f4a63409004abea7b3f26310cab611"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "d8d2a52861a89bbc48382ea1aad5b127"
server_key_iv: "44a2929a980ac0a4eba266e4f4d32c50"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.425204 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:24:50.430420 21001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:50.432441 20987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:24:50.432734 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "50f4a63409004abea7b3f26310cab611"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "d8d2a52861a89bbc48382ea1aad5b127"
server_key_iv: "44a2929a980ac0a4eba266e4f4d32c50"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.433211 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:50.463364 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:50.464684 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:50.465097 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:50.466737 20987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:50.469672 20987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:50.469810 20987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:50.470031 20987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:50.470129 20987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:50.517551 20987 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:43341
I20260927 15:24:50.517621 21113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:43341 every 8 connection(s)
I20260927 15:24:50.519678 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:50.528074 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 20987
I20260927 15:24:50.528398 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:24:50.529047 3033 external_mini_cluster.cc:1428] Setting key f2f88f024b82b1966212048b80ff9b0d
I20260927 15:24:50.533452 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--builtin_ntp_servers=127.2.246.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:24:50.537251 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:50.523159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:37351 (local address 127.2.246.126:38817)
0927 15:24:50.523673 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:50.523677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:50.525390 (+ 1713us) server_negotiation.cc:407] Connection header received
0927 15:24:50.527510 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:50.527517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:50.527618 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:50.527771 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:50.529116 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:50.530013 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:50.532900 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:50.533642 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:50.533875 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:50.536021 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:50.536031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:50.536038 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:50.536072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:50.536134 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:50.536156 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:50.536292 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:50.536626 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:24:50.536768 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":298,"threads_started":1}
I20260927 15:24:50.538905 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:50.523832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38817 (local address 127.2.246.66:37351)
0927 15:24:50.525165 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:50.525220 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:50.527152 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:50.528038 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:50.528049 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:50.528158 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:50.528923 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:50.528941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:50.530288 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:50.530295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:50.532589 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:50.532619 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:50.534083 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:50.534090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:50.534424 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:50.535418 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:50.535422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:50.535482 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:50.535615 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:50.536333 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:50.536350 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:50.536361 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:24:50.536578 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:24:50.537074 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":430,"threads_started":1}
I20260927 15:24:50.541172 21114 heartbeater.cc:347] Connected to a master server at 127.2.246.126:38817
I20260927 15:24:50.541646 21114 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:50.542632 21114 heartbeater.cc:511] Master 127.2.246.126:38817 requested a full tablet report, sending...
I20260927 15:24:50.544772 20797 ts_manager.cc:194] Registered new tserver with Master: 50f4a63409004abea7b3f26310cab611 (127.2.246.66:43341)
I20260927 15:24:50.546136 20797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.66:37351
W20260927 15:24:50.888007 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:50.888379 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:50.888521 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:50.896189 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:50.896364 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:50.896524 21118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:50.906656 21118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:50.908051 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:50.909960 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:50.924719 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:50.924719 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:50.924628 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:50.926530 21118 server_base.cc:1034] running on GCE node
I20260927 15:24:50.927181 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:50.928295 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:50.929522 21118 hybrid_clock.cc:648] HybridClock initialized: now 1790522690929468 us; error 53 us; skew 500 ppm
I20260927 15:24:50.930016 21118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:50.932806 21118 webserver.cc:460] Webserver started at http://127.2.246.67:45887/ using document root <none> and password file <none>
I20260927 15:24:50.933641 21118 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:50.933785 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:50.934159 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:50.937254 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "12e9dc9cf6d74631b5f4b3058c2918bd"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "a5ca45c674ca268a5ed68e6cd2f2eb4b"
server_key_iv: "4def2dea2d99a84731f311ae5261ff8d"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.938269 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "12e9dc9cf6d74631b5f4b3058c2918bd"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "a5ca45c674ca268a5ed68e6cd2f2eb4b"
server_key_iv: "4def2dea2d99a84731f311ae5261ff8d"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.944509 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:24:50.949280 21132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:50.951277 21118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:24:50.951520 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "12e9dc9cf6d74631b5f4b3058c2918bd"
format_stamp: "Formatted at 2026-09-27 15:24:50 on dist-test-slave-14jq"
server_key: "a5ca45c674ca268a5ed68e6cd2f2eb4b"
server_key_iv: "4def2dea2d99a84731f311ae5261ff8d"
server_key_version: "encryptionkey@0"
I20260927 15:24:50.951824 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:50.988476 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:50.989387 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:50.989705 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:50.990890 21118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:50.992513 21118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:50.992664 21118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:50.992805 21118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:50.992897 21118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:51.003990 20983 heartbeater.cc:503] Master 127.2.246.126:38817 was elected leader, sending a full tablet report...
I20260927 15:24:51.033213 21118 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:35439
I20260927 15:24:51.033264 21244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:35439 every 8 connection(s)
I20260927 15:24:51.035164 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:51.035770 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21118
I20260927 15:24:51.036018 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:24:51.036816 3033 external_mini_cluster.cc:1428] Setting key 8fe06fec5ee00ca074fca446f8d8c161
I20260927 15:24:51.052460 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.039202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:37473 (local address 127.2.246.126:38817)
0927 15:24:51.039963 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.039972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.041677 (+ 1705us) server_negotiation.cc:407] Connection header received
0927 15:24:51.044050 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.044056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.044152 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.044297 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:51.045652 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.046677 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.048602 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.049399 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.049571 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.051338 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:51.051345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:51.051349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:51.051377 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:51.051437 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:51.051453 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:51.051564 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.051899 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.052027 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":597,"threads_started":1}
I20260927 15:24:51.053516 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.040292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38817 (local address 127.2.246.67:37473)
0927 15:24:51.041480 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.041523 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.043698 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.044531 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.044540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.044719 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:51.045441 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.045459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.046862 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.046866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.048348 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.048369 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.049766 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.049773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.050107 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.050924 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:51.050927 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:51.050973 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:51.051065 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:51.051606 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:51.051618 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:51.051627 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.051833 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.052199 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":480,"threads_started":1}
I20260927 15:24:51.055642 21245 heartbeater.cc:347] Connected to a master server at 127.2.246.126:38817
I20260927 15:24:51.055996 21245 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:51.057024 21245 heartbeater.cc:511] Master 127.2.246.126:38817 requested a full tablet report, sending...
I20260927 15:24:51.058825 20797 ts_manager.cc:194] Registered new tserver with Master: 12e9dc9cf6d74631b5f4b3058c2918bd (127.2.246.67:35439)
I20260927 15:24:51.059868 20797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.67:37473
I20260927 15:24:51.071437 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:24:51.084688 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.077067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.2.246.126:38817)
0927 15:24:51.077416 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.077424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.077802 (+ 378us) server_negotiation.cc:407] Connection header received
0927 15:24:51.078189 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.078195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.078278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.078396 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:51.078964 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.079768 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.081527 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.082089 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.082250 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.083616 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:51.083621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:51.083625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:51.083650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:51.083703 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:51.083718 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:51.083826 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.084067 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.084320 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:24:51.090411 20797 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:24:51.099887 20797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:24:51.126118 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.115916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.2.246.65:42991)
0927 15:24:51.116516 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.116523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.116760 (+ 237us) server_negotiation.cc:407] Connection header received
0927 15:24:51.117143 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.117155 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.117482 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.117706 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:51.118675 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.120171 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.123217 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.124618 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.124781 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.125089 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.125519 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.125703 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":300,"threads_started":1}
I20260927 15:24:51.126188 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.115665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52922)
0927 15:24:51.116567 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.116619 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.116841 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.118023 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.118033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.118076 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:51.118461 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.118479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.120362 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.120368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.122949 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.122975 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.124875 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.124882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.125196 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.125271 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.125443 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.125697 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":405,"threads_started":1}
I20260927 15:24:51.129318 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.117055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35439 (local address 127.0.0.1:58490)
0927 15:24:51.117741 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.117764 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.117916 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.121188 (+ 3272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.121192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.121217 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:51.121937 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.121951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.124127 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.124133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.126344 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.126363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.128411 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.128419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.128630 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.128672 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.128769 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.128921 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":1,"thread_start_us":357,"threads_started":1}
I20260927 15:24:51.130411 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.117749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43341 (local address 127.0.0.1:55490)
0927 15:24:51.121572 (+ 3823us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.121624 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.121806 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.122494 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.122499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.122528 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:51.123014 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.123032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.124856 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.124862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.127197 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.127214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.129070 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.129076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.129278 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.129316 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.129431 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.129592 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3691,"thread_start_us":3470,"threads_started":1}
I20260927 15:24:51.130609 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.117662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.2.246.67:35439)
0927 15:24:51.120598 (+ 2936us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.120606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.120640 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:24:51.120742 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.120749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.120974 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.121632 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:51.122192 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.123950 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.126626 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.128161 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.129491 (+ 1330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.129806 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.129943 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.130176 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2748,"thread_start_us":2587,"threads_started":1}
I20260927 15:24:51.131121 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.118335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.2.246.66:43341)
0927 15:24:51.120977 (+ 2642us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.120989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.121940 (+ 951us) server_negotiation.cc:407] Connection header received
0927 15:24:51.122055 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.122062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.122262 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.122458 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:51.123377 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.124652 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.127454 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.128837 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.129018 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.129336 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.130173 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.130413 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2436,"thread_start_us":2292,"threads_started":1}
I20260927 15:24:51.130885 20918 tablet_service.cc:1467] Processing CreateTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 (DEFAULT_TABLE table=test-table [id=430abbb4b8ec47d2b96e5f67cc65fa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:51.132385 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:51.135531 21049 tablet_service.cc:1467] Processing CreateTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 (DEFAULT_TABLE table=test-table [id=430abbb4b8ec47d2b96e5f67cc65fa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:51.136812 21049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:51.136538 21180 tablet_service.cc:1467] Processing CreateTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 (DEFAULT_TABLE table=test-table [id=430abbb4b8ec47d2b96e5f67cc65fa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:51.138561 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:51.149729 21264 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Bootstrap starting.
I20260927 15:24:51.153644 21264 tablet_bootstrap.cc:654] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:51.154783 21265 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Bootstrap starting.
I20260927 15:24:51.155081 21264 log.cc:824] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:51.158787 21267 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Bootstrap starting.
I20260927 15:24:51.160624 21265 tablet_bootstrap.cc:654] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:51.161988 21265 log.cc:824] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:51.163848 21267 tablet_bootstrap.cc:654] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:51.164924 21264 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: No bootstrap required, opened a new log
I20260927 15:24:51.165227 21267 log.cc:824] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:51.165303 21264 ts_tablet_manager.cc:1397] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260927 15:24:51.169138 21267 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: No bootstrap required, opened a new log
I20260927 15:24:51.169488 21267 ts_tablet_manager.cc:1397] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 15:24:51.170011 21265 tablet_bootstrap.cc:492] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: No bootstrap required, opened a new log
I20260927 15:24:51.170359 21265 ts_tablet_manager.cc:1397] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260927 15:24:51.176465 21264 raft_consensus.cc:348] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.177026 21264 raft_consensus.cc:374] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:51.177121 21264 raft_consensus.cc:729] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f4a63409004abea7b3f26310cab611, State: Initialized, Role: FOLLOWER
I20260927 15:24:51.178020 21264 consensus_queue.cc:260] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.179646 21114 heartbeater.cc:503] Master 127.2.246.126:38817 was elected leader, sending a full tablet report...
I20260927 15:24:51.180303 21264 ts_tablet_manager.cc:1428] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260927 15:24:51.180891 21265 raft_consensus.cc:348] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.181562 21265 raft_consensus.cc:374] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:51.181699 21265 raft_consensus.cc:729] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3951c406c95d45c582bba127513d27c3, State: Initialized, Role: FOLLOWER
I20260927 15:24:51.182710 21265 consensus_queue.cc:260] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.183272 21267 raft_consensus.cc:348] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.183909 21267 raft_consensus.cc:374] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:51.184036 21267 raft_consensus.cc:729] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9dc9cf6d74631b5f4b3058c2918bd, State: Initialized, Role: FOLLOWER
I20260927 15:24:51.184916 21267 consensus_queue.cc:260] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.185164 21265 ts_tablet_manager.cc:1428] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 15:24:51.186632 21245 heartbeater.cc:503] Master 127.2.246.126:38817 was elected leader, sending a full tablet report...
I20260927 15:24:51.187295 21267 ts_tablet_manager.cc:1428] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260927 15:24:51.197242 21270 raft_consensus.cc:482] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:51.197676 21270 raft_consensus.cc:504] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.200089 21270 leader_election.cc:288] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3951c406c95d45c582bba127513d27c3 (127.2.246.65:42991), 12e9dc9cf6d74631b5f4b3058c2918bd (127.2.246.67:35439)
I20260927 15:24:51.208820 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.200770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:41057 (local address 127.2.246.65:42991)
0927 15:24:51.201080 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.201086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.201600 (+ 514us) server_negotiation.cc:407] Connection header received
0927 15:24:51.201936 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.201944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.202048 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.202124 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:51.203105 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.204165 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.206625 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.207719 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.207938 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.208031 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.208347 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.208508 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 15:24:51.208985 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.200629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.2.246.66:41057)
0927 15:24:51.201457 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.201478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.201642 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.202368 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.202371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.202393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:51.202912 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.202927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.204368 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.204373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.206393 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.206410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.207954 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.207960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.208160 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.208191 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.208312 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.208485 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":131,"spinlock_wait_cycles":42880,"thread_start_us":454,"threads_started":1}
I20260927 15:24:51.209530 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.201891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:42309 (local address 127.2.246.67:35439)
0927 15:24:51.202245 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.202252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.202469 (+ 217us) server_negotiation.cc:407] Connection header received
0927 15:24:51.202689 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.202696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.202790 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.202918 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:51.203668 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.204659 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.207031 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.208259 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.208449 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.208521 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.209124 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.209262 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260927 15:24:51.210315 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.201531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:35439 (local address 127.2.246.66:42309)
0927 15:24:51.202318 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.202343 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:51.202529 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:51.202956 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:51.202960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:51.203025 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:51.203469 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.203482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.204887 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.204892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.206802 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:51.206821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.208664 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.208671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:51.208893 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.208942 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 15:24:51.209752 (+ 810us) client_negotiation.cc:240] Negotiation successful
0927 15:24:51.209945 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":1,"thread_start_us":354,"threads_started":1}
I20260927 15:24:51.210512 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea0050d6ceb4e1b8ceab6e94e24c0a8" candidate_uuid: "50f4a63409004abea7b3f26310cab611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3951c406c95d45c582bba127513d27c3" is_pre_election: true
I20260927 15:24:51.211261 20938 raft_consensus.cc:2454] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f4a63409004abea7b3f26310cab611 in term 0.
I20260927 15:24:51.211457 21200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea0050d6ceb4e1b8ceab6e94e24c0a8" candidate_uuid: "50f4a63409004abea7b3f26310cab611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" is_pre_election: true
I20260927 15:24:51.212047 21200 raft_consensus.cc:2454] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f4a63409004abea7b3f26310cab611 in term 0.
I20260927 15:24:51.212213 21005 leader_election.cc:302] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3951c406c95d45c582bba127513d27c3, 50f4a63409004abea7b3f26310cab611; no voters:
I20260927 15:24:51.212829 21270 raft_consensus.cc:2781] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:51.213014 21270 raft_consensus.cc:482] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:51.213117 21270 raft_consensus.cc:3037] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:51.216403 21270 raft_consensus.cc:504] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
I20260927 15:24:51.217237 21270 leader_election.cc:288] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [CANDIDATE]: Term 1 election: Requested vote from peers 3951c406c95d45c582bba127513d27c3 (127.2.246.65:42991), 12e9dc9cf6d74631b5f4b3058c2918bd (127.2.246.67:35439)
I20260927 15:24:51.218335 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea0050d6ceb4e1b8ceab6e94e24c0a8" candidate_uuid: "50f4a63409004abea7b3f26310cab611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3951c406c95d45c582bba127513d27c3"
I20260927 15:24:51.218418 21200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea0050d6ceb4e1b8ceab6e94e24c0a8" candidate_uuid: "50f4a63409004abea7b3f26310cab611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd"
I20260927 15:24:51.218695 20938 raft_consensus.cc:3037] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:51.218695 21200 raft_consensus.cc:3037] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:51.223269 20938 raft_consensus.cc:2454] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f4a63409004abea7b3f26310cab611 in term 1.
I20260927 15:24:51.223269 21200 raft_consensus.cc:2454] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f4a63409004abea7b3f26310cab611 in term 1.
I20260927 15:24:51.224028 21005 leader_election.cc:302] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3951c406c95d45c582bba127513d27c3, 50f4a63409004abea7b3f26310cab611; no voters:
I20260927 15:24:51.224488 21270 raft_consensus.cc:2781] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:51.225114 21270 raft_consensus.cc:686] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 1 LEADER]: Becoming Leader. State: Replica: 50f4a63409004abea7b3f26310cab611, State: Running, Role: LEADER
I20260927 15:24:51.225728 21270 consensus_queue.cc:237] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } }
W20260927 15:24:51.227144 20984 tablet.cc:2369] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:51.231506 20796 catalog_manager.cc:5495] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50f4a63409004abea7b3f26310cab611 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "50f4a63409004abea7b3f26310cab611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50f4a63409004abea7b3f26310cab611" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43341 } health_report { overall_health: HEALTHY } } }
I20260927 15:24:51.259176 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.252450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.2.246.66:43341)
0927 15:24:51.252670 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.252676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.252788 (+ 112us) server_negotiation.cc:407] Connection header received
0927 15:24:51.253029 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.253033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.253137 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.253314 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:51.253765 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.254623 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.256521 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.257145 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.257284 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.258016 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:51.258569 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:24:51.258708 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.258870 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.258949 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:24:51.270509 21200 raft_consensus.cc:1261] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 1 FOLLOWER]: Refusing update from remote peer 50f4a63409004abea7b3f26310cab611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:51.270773 20938 raft_consensus.cc:1261] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3 [term 1 FOLLOWER]: Refusing update from remote peer 50f4a63409004abea7b3f26310cab611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:51.271869 21275 consensus_queue.cc:1035] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9dc9cf6d74631b5f4b3058c2918bd" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:24:51.272477 21115 tablet.cc:2369] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:51.272476 21270 consensus_queue.cc:1035] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3951c406c95d45c582bba127513d27c3" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:51.286109 21278 mvcc.cc:205] Tried to move back new op lower bound from 7333980943430787072 to 7333980943266185216. Current Snapshot: MvccSnapshot[applied={T|T < 7333980943430787072}]
I20260927 15:24:51.305472 20797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:51.305842 20797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:51.310550 20797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 599ef66385da4359b2181d629b14c638: Sending DeleteTablet for 3 replicas of tablet cea0050d6ceb4e1b8ceab6e94e24c0a8
I20260927 15:24:51.312182 20918 tablet_service.cc:1514] Processing DeleteTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:51 UTC) from {username='slave'} at 127.0.0.1:52922
I20260927 15:24:51.312542 21180 tablet_service.cc:1514] Processing DeleteTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:51 UTC) from {username='slave'} at 127.0.0.1:58490
I20260927 15:24:51.312513 21049 tablet_service.cc:1514] Processing DeleteTablet for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:51 UTC) from {username='slave'} at 127.0.0.1:55490
I20260927 15:24:51.313690 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20856
I20260927 15:24:51.333268 21288 tablet_replica.cc:330] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: stopping tablet replica
I20260927 15:24:51.335804 21288 raft_consensus.cc:2229] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:24:51.338925 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.315437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.2.246.65:42991)
0927 15:24:51.315688 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.315695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.326648 (+ 10953us) server_negotiation.cc:407] Connection header received
0927 15:24:51.327300 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.327309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.327468 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.327565 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:51.328476 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.330084 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.333530 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.334267 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.334377 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.336526 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:51.336538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:51.336558 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:51.337405 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:51.337476 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:51.337504 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:51.337594 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.338352 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.338498 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:24:51.340941 21288 raft_consensus.cc:2258] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:51.341109 20978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:51.342264 21289 tablet_replica.cc:330] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: stopping tablet replica
I20260927 15:24:51.342830 21289 raft_consensus.cc:2229] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:51.343493 21289 raft_consensus.cc:2258] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:51.346438 21288 ts_tablet_manager.cc:1905] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:51.353497 21289 ts_tablet_manager.cc:1905] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:51.354563 21288 ts_tablet_manager.cc:1918] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:51.354877 21288 log.cc:1196] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/cea0050d6ceb4e1b8ceab6e94e24c0a8
I20260927 15:24:51.355624 21288 ts_tablet_manager.cc:1939] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 50f4a63409004abea7b3f26310cab611: Deleting consensus metadata
I20260927 15:24:51.357980 20784 catalog_manager.cc:4841] TS 50f4a63409004abea7b3f26310cab611 (127.2.246.66:43341): tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 (table test-table [id=430abbb4b8ec47d2b96e5f67cc65fa4b]) successfully deleted
I20260927 15:24:51.362644 21289 ts_tablet_manager.cc:1918] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:51.362916 21289 log.cc:1196] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/cea0050d6ceb4e1b8ceab6e94e24c0a8
I20260927 15:24:51.363739 21289 ts_tablet_manager.cc:1939] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 12e9dc9cf6d74631b5f4b3058c2918bd: Deleting consensus metadata
I20260927 15:24:51.366606 20784 catalog_manager.cc:4841] TS 12e9dc9cf6d74631b5f4b3058c2918bd (127.2.246.67:35439): tablet cea0050d6ceb4e1b8ceab6e94e24c0a8 (table test-table [id=430abbb4b8ec47d2b96e5f67cc65fa4b]) successfully deleted
I20260927 15:24:51.454245 21287 tablet_replica.cc:330] T cea0050d6ceb4e1b8ceab6e94e24c0a8 P 3951c406c95d45c582bba127513d27c3: stopping tablet replica
I20260927 15:24:51.454373 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20856
I20260927 15:24:51.480435 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20987
W20260927 15:24:51.480362 20784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:42991 (error 108)
I20260927 15:24:51.483819 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.483265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52946)
0927 15:24:51.483505 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.483629 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 15:24:51.484184 20784 catalog_manager.cc:4584] TS 3951c406c95d45c582bba127513d27c3 (127.2.246.65:42991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cea0050d6ceb4e1b8ceab6e94e24c0a8: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
I20260927 15:24:51.490315 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.481520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.2.246.66:43341)
0927 15:24:51.481754 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.481760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.482267 (+ 507us) server_negotiation.cc:407] Connection header received
0927 15:24:51.482347 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.482353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.482457 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.482551 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:51.483147 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.484492 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.486419 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.487087 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.487234 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.488527 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:51.488538 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:51.488558 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:51.489402 (+ 844us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:51.489472 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:51.489496 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:51.489612 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.489782 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.489908 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:24:51.491442 21109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:51.538455 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.537670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52954)
0927 15:24:51.538058 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.538206 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 15:24:51.540156 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.539569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52962)
0927 15:24:51.539836 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.539973 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 15:24:51.601496 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.600913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52972)
0927 15:24:51.601159 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.601260 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 15:24:51.602650 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.602296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52974)
0927 15:24:51.602450 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.602534 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:24:51.615592 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20987
I20260927 15:24:51.637380 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21118
I20260927 15:24:51.646991 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.638398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.2.246.67:35439)
0927 15:24:51.638695 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:51.638700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:51.638725 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:24:51.638831 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:51.638837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:51.638963 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:51.639057 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:51.639546 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.640520 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.642637 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:51.643325 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:51.643472 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:51.644879 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:51.644890 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:51.644922 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:51.645729 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:51.645806 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:51.645949 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:51.646140 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:51.646367 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:24:51.646517 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 15:24:51.648017 21240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:51.683818 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.683220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52976)
0927 15:24:51.683504 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.683608 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 15:24:51.685132 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:51.684628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42991 (local address 127.0.0.1:52988)
0927 15:24:51.684878 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:51.684960 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 15:24:51.740934 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21118
I20260927 15:24:51.762658 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 20766
I20260927 15:24:51.764091 20827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:51.878935 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 20766
2026-09-27T15:24:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3279 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:24:52 dist-test-slave-14jq krb5kdc[21298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:24:52 dist-test-slave-14jq krb5kdc[21298](info): set up 2 sockets
Sep 27 15:24:52 dist-test-slave-14jq krb5kdc[21298](info): commencing operation
krb5kdc: starting...
W20260927 15:24:53.078541 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:24:53 dist-test-slave-14jq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:24:53.381655 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42153
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:53.746752 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:53.747155 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:53.747334 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:53.755304 21311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:53.755467 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:53.755581 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:53.755692 21311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:53.755769 21311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:53.766465 21311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:53.767918 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:53.769250 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:53.783170 21319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:53.783466 21316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:53.783555 21317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:53.784847 21311 server_base.cc:1034] running on GCE node
I20260927 15:24:53.785941 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:53.787678 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:53.789000 21311 hybrid_clock.cc:648] HybridClock initialized: now 1790522693788953 us; error 61 us; skew 500 ppm
I20260927 15:24:53.789464 21311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:53 dist-test-slave-14jq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:53.793491 21311 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:24:53.795603 21311 webserver.cc:460] Webserver started at http://127.2.246.126:41499/ using document root <none> and password file <none>
I20260927 15:24:53.796482 21311 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:53.796608 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:53.796936 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:53.800776 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "6fdd23bcb8f84c3ba8377f17ab754abe"
format_stamp: "Formatted at 2026-09-27 15:24:53 on dist-test-slave-14jq"
server_key: "8795fd705ba4252fd7afde81abc40709"
server_key_iv: "5072a463fea8e11f8c7aa21a17867925"
server_key_version: "encryptionkey@0"
I20260927 15:24:53.801667 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "6fdd23bcb8f84c3ba8377f17ab754abe"
format_stamp: "Formatted at 2026-09-27 15:24:53 on dist-test-slave-14jq"
server_key: "8795fd705ba4252fd7afde81abc40709"
server_key_iv: "5072a463fea8e11f8c7aa21a17867925"
server_key_version: "encryptionkey@0"
I20260927 15:24:53.808935 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:24:53.814646 21326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:53.816766 21311 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:24:53.817059 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "6fdd23bcb8f84c3ba8377f17ab754abe"
format_stamp: "Formatted at 2026-09-27 15:24:53 on dist-test-slave-14jq"
server_key: "8795fd705ba4252fd7afde81abc40709"
server_key_iv: "5072a463fea8e11f8c7aa21a17867925"
server_key_version: "encryptionkey@0"
I20260927 15:24:53.817371 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:53.844240 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:53.850447 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:53.850842 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:53.876631 21311 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:42153
I20260927 15:24:53.876609 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:42153 every 8 connection(s)
I20260927 15:24:53.878926 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:53.883805 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:53.887868 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21311
I20260927 15:24:53.888131 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:24:53.888648 3033 external_mini_cluster.cc:1428] Setting key adbfd75a718e0f05fd85f4ab81ee2d23
I20260927 15:24:53.896123 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: Bootstrap starting.
I20260927 15:24:53.900059 21378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:53.901463 21378 log.cc:824] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:53.906054 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: No bootstrap required, opened a new log
Sep 27 15:24:53 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:53.917279 21378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } }
I20260927 15:24:53.917809 21378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:53.918004 21378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdd23bcb8f84c3ba8377f17ab754abe, State: Initialized, Role: FOLLOWER
I20260927 15:24:53.918802 21378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } }
I20260927 15:24:53.919119 21378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:53.919275 21378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:53.919483 21378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:53.920697 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:53.891025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.2.246.126:42153)
0927 15:24:53.892225 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:53.892243 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:53.892309 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:24:53.895032 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:53.895071 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:53.895699 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:53.896176 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:53.898304 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:53.899886 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:53.902028 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:53.903526 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:53.903745 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:53.909594 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:53.909612 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:53.909635 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:53.909697 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:53.914240 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:53.915462 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:53.915473 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:53.915484 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:53.915781 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:53.916326 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:53.916332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:53.916344 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:53.917164 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:53.917506 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:53.918371 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 15:24:53.918982 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":591,"threads_started":1}
I20260927 15:24:53.922573 21378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } }
I20260927 15:24:53.923275 21378 leader_election.cc:302] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fdd23bcb8f84c3ba8377f17ab754abe; no voters:
I20260927 15:24:53.924139 21378 leader_election.cc:288] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:24:53.924283 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:53.925076 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [term 1 LEADER]: Becoming Leader. State: Replica: 6fdd23bcb8f84c3ba8377f17ab754abe, State: Running, Role: LEADER
I20260927 15:24:53.925987 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } }
I20260927 15:24:53.927387 21378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:53.929562 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } } }
I20260927 15:24:53.929447 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fdd23bcb8f84c3ba8377f17ab754abe. Latest consensus state: current_term: 1 leader_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdd23bcb8f84c3ba8377f17ab754abe" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42153 } } }
I20260927 15:24:53.930331 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:53.930369 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:53.931566 21388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:53.939637 21388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:53.955760 21388 catalog_manager.cc:1347] Generated new cluster ID: 18de1e2feab4448cac825e6011a15e01
I20260927 15:24:53.955950 21388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:53.976279 21388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:24:53.978072 21388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:53.986091 21388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: Generated new TSK 0
I20260927 15:24:53.987484 21388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:24:54.070851 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:54.506695 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:54.507087 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:54.507200 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:54.516711 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:54.516919 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:54.517077 21404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:54.530323 21404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:54.532099 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:54.533877 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:54.551472 21409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:54.551476 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:54.551663 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:54.553000 21404 server_base.cc:1034] running on GCE node
I20260927 15:24:54.553977 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:54.555356 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:54.556671 21404 hybrid_clock.cc:648] HybridClock initialized: now 1790522694556560 us; error 81 us; skew 500 ppm
I20260927 15:24:54.557109 21404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:54 dist-test-slave-14jq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:54.561170 21404 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:24:54.563344 21404 webserver.cc:460] Webserver started at http://127.2.246.65:45141/ using document root <none> and password file <none>
I20260927 15:24:54.564245 21404 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:54.564368 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:54.564708 21404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:54.568528 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "928d0e1d81784dc6ac4b00c7a91a124f"
format_stamp: "Formatted at 2026-09-27 15:24:54 on dist-test-slave-14jq"
server_key: "76488af2e3085300bdaff6acc36ef593"
server_key_iv: "6ef3f4cd9be2a3a33b59c1c7094fc6b4"
server_key_version: "encryptionkey@0"
I20260927 15:24:54.569564 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "928d0e1d81784dc6ac4b00c7a91a124f"
format_stamp: "Formatted at 2026-09-27 15:24:54 on dist-test-slave-14jq"
server_key: "76488af2e3085300bdaff6acc36ef593"
server_key_iv: "6ef3f4cd9be2a3a33b59c1c7094fc6b4"
server_key_version: "encryptionkey@0"
I20260927 15:24:54.578135 21404 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 15:24:54.583261 21419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:54.585736 21404 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:24:54.586138 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "928d0e1d81784dc6ac4b00c7a91a124f"
format_stamp: "Formatted at 2026-09-27 15:24:54 on dist-test-slave-14jq"
server_key: "76488af2e3085300bdaff6acc36ef593"
server_key_iv: "6ef3f4cd9be2a3a33b59c1c7094fc6b4"
server_key_version: "encryptionkey@0"
I20260927 15:24:54.586429 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:54.611276 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:54.617697 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:54.618180 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:54.619647 21404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:54.621582 21404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:54.621675 21404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:54.621835 21404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:54.621929 21404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:24:54.664177 21404 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:35195
I20260927 15:24:54.664237 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:35195 every 8 connection(s)
I20260927 15:24:54.665988 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:54.668627 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21404
I20260927 15:24:54.668876 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:24:54.669394 3033 external_mini_cluster.cc:1428] Setting key 5c62a0d8c922792a9785dc86e944dfb9
Sep 27 15:24:54 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:54.694836 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:54.669930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:46695 (local address 127.2.246.126:42153)
0927 15:24:54.670532 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:54.670539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:54.672097 (+ 1558us) server_negotiation.cc:407] Connection header received
0927 15:24:54.674624 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:54.674630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:54.674737 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:54.674859 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:54.676551 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:54.677442 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:54.680842 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:54.681291 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:54.681422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:54.687227 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:54.687234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:54.687238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:54.687287 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:54.690501 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:54.691810 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:54.691814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:54.691817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:54.691966 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:54.692562 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:54.692566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:54.692568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:54.692879 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:54.693034 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:54.694254 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0927 15:24:54.694521 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
I20260927 15:24:54.696362 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:54.670758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42153 (local address 127.2.246.65:46695)
0927 15:24:54.671889 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:54.671965 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:54.674308 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:54.675141 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:54.675150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:54.675747 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:54.676365 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:54.676382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:54.677759 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:54.677770 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:54.680507 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:54.680543 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:54.681653 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:54.681664 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:54.682138 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:54.683181 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:54.683223 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:54.686835 (+ 3612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:54.690734 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:54.690749 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:54.690773 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:54.691588 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:54.692108 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:54.692116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:54.692122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:54.692420 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:54.693025 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:54.693036 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:54.693562 (+ 526us) client_negotiation.cc:769] Sending connection context
0927 15:24:54.694051 (+ 489us) client_negotiation.cc:240] Negotiation successful
0927 15:24:54.694703 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":412,"threads_started":1}
I20260927 15:24:54.698676 21532 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42153
I20260927 15:24:54.699183 21532 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:54.700309 21532 heartbeater.cc:511] Master 127.2.246.126:42153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:24:54.703588 21343 ts_manager.cc:194] Registered new tserver with Master: 928d0e1d81784dc6ac4b00c7a91a124f (127.2.246.65:35195)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
I20260927 15:24:54.705621 21343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:46695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:24:54.739434 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:24:55.106220 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:55.106586 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:55.106755 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:55.114435 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:55.114610 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:55.114802 21538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:55.126084 21538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:55.127522 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:55.129065 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:55.145673 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:55.145954 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:55.145658 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:55.146718 21538 server_base.cc:1034] running on GCE node
I20260927 15:24:55.147593 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:55.148819 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:55.150166 21538 hybrid_clock.cc:648] HybridClock initialized: now 1790522695150081 us; error 65 us; skew 500 ppm
I20260927 15:24:55.150568 21538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:55 dist-test-slave-14jq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:55.154435 21538 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:24:55.156428 21538 webserver.cc:460] Webserver started at http://127.2.246.66:43839/ using document root <none> and password file <none>
I20260927 15:24:55.157207 21538 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:55.157368 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:55.157708 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:55.160869 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "0d4751a29c4145fd8ff01629d3aacf7c"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "790e63a09725297f82d1c9f18e7a5197"
server_key_iv: "fd9c0110ed2d79c6337d10b03ff146ab"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.161755 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "0d4751a29c4145fd8ff01629d3aacf7c"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "790e63a09725297f82d1c9f18e7a5197"
server_key_iv: "fd9c0110ed2d79c6337d10b03ff146ab"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.168428 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:24:55.173152 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.175252 21538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:24:55.175521 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "0d4751a29c4145fd8ff01629d3aacf7c"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "790e63a09725297f82d1c9f18e7a5197"
server_key_iv: "fd9c0110ed2d79c6337d10b03ff146ab"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.175793 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:55.196170 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:55.201331 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:55.201671 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:55.202979 21538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:55.204584 21538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:55.204697 21538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.204871 21538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:55.204952 21538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.249140 21538 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:41919
I20260927 15:24:55.249179 21665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:41919 every 8 connection(s)
I20260927 15:24:55.250800 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:55.251871 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21538
I20260927 15:24:55.252112 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:24:55.252790 3033 external_mini_cluster.cc:1428] Setting key 5324498abd0f0355a8fbe3dba4507bbd
Sep 27 15:24:55 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:55.287032 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:55.254651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:39413 (local address 127.2.246.126:42153)
0927 15:24:55.255316 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:55.255326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:55.257023 (+ 1697us) server_negotiation.cc:407] Connection header received
0927 15:24:55.261287 (+ 4264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:55.261293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:55.261396 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:55.261546 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:55.265105 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.266128 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.270398 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.271340 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.271538 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:55.276855 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:55.276868 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:55.276877 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:55.276945 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:55.281613 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:55.283050 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:55.283059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:55.283066 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:55.283277 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:55.284058 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:55.284066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:55.284073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:55.284571 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:55.284774 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:55.286112 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0927 15:24:55.286469 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":329,"threads_started":1}
I20260927 15:24:55.288105 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:55.255389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42153 (local address 127.2.246.66:39413)
0927 15:24:55.256746 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:55.256840 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:55.260599 (+ 3759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:55.262106 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:55.262128 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:55.263341 (+ 1213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:55.264665 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:55.264720 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.266416 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.266428 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:55.270056 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:55.270102 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.271924 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.271942 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:55.272206 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:55.273168 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:55.273203 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:55.276387 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:55.282005 (+ 5618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:55.282018 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:55.282036 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:55.282686 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:55.283489 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:55.283495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:55.283498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:55.283871 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:55.284791 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:55.284804 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:55.285437 (+ 633us) client_negotiation.cc:769] Sending connection context
0927 15:24:55.285953 (+ 516us) client_negotiation.cc:240] Negotiation successful
0927 15:24:55.286566 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":516,"threads_started":1}
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:24:55.290527 21666 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42153
I20260927 15:24:55.290954 21666 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:55.292019 21666 heartbeater.cc:511] Master 127.2.246.126:42153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
I20260927 15:24:55.294517 21343 ts_manager.cc:194] Registered new tserver with Master: 0d4751a29c4145fd8ff01629d3aacf7c (127.2.246.66:41919)
I20260927 15:24:55.295619 21343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:39413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:24:55.329547 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:24:55.711244 21532 heartbeater.cc:503] Master 127.2.246.126:42153 was elected leader, sending a full tablet report...
W20260927 15:24:55.765708 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:55.766212 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:55.766330 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:55.775870 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:55.776057 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:55.776204 21672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:55.790017 21672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:55.791898 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:55.793644 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:55.812004 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:55.812088 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:55.812088 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:55.813467 21672 server_base.cc:1034] running on GCE node
I20260927 15:24:55.814318 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:55.815461 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:55.816732 21672 hybrid_clock.cc:648] HybridClock initialized: now 1790522695816674 us; error 47 us; skew 500 ppm
I20260927 15:24:55.817149 21672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:24:55 dist-test-slave-14jq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:24:55.821463 21672 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:24:55.823786 21672 webserver.cc:460] Webserver started at http://127.2.246.67:38239/ using document root <none> and password file <none>
I20260927 15:24:55.824647 21672 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:55.824826 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:55.825214 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:55.828662 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "9ff9e2fcdf9745758ae1ef19a574ff14"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "cebd472de29507a5da35cc5178a40ccc"
server_key_iv: "1c646eb3be3aec74f1e2cd3533af731e"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.829596 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "9ff9e2fcdf9745758ae1ef19a574ff14"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "cebd472de29507a5da35cc5178a40ccc"
server_key_iv: "1c646eb3be3aec74f1e2cd3533af731e"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.835933 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 15:24:55.840607 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.842730 21672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:24:55.842926 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "9ff9e2fcdf9745758ae1ef19a574ff14"
format_stamp: "Formatted at 2026-09-27 15:24:55 on dist-test-slave-14jq"
server_key: "cebd472de29507a5da35cc5178a40ccc"
server_key_iv: "1c646eb3be3aec74f1e2cd3533af731e"
server_key_version: "encryptionkey@0"
I20260927 15:24:55.843135 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:55.893357 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:55.897874 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:55.898132 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:55.899341 21672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:55.901039 21672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:55.901154 21672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.901335 21672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:55.901427 21672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:55.941568 21672 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:37857
I20260927 15:24:55.941619 21799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:37857 every 8 connection(s)
I20260927 15:24:55.943154 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:55.953291 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21672
I20260927 15:24:55.953522 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:24:55.954144 3033 external_mini_cluster.cc:1428] Setting key e4976d07c8bf2d8ff01fe67b528e26e6
Sep 27 15:24:55 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:24:55.969470 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:55.946949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:49531 (local address 127.2.246.126:42153)
0927 15:24:55.947649 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:55.947657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:55.949208 (+ 1551us) server_negotiation.cc:407] Connection header received
0927 15:24:55.951432 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:55.951438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:55.951556 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:55.951724 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:55.953999 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.955132 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.957139 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.957693 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.957843 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:55.962556 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:55.962562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:55.962566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:55.962621 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:55.965468 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:55.966478 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:55.966483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:55.966486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:55.966643 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:55.967138 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:55.967141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:55.967144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:55.967477 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:55.967617 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:55.968683 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0927 15:24:55.968981 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":310,"threads_started":1}
I20260927 15:24:55.970762 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:55.947790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42153 (local address 127.2.246.67:49531)
0927 15:24:55.948982 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:55.949035 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:55.951050 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:55.952059 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:55.952069 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:55.952702 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:55.953688 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:55.953717 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.955402 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.955407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:55.956938 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:55.956959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.958003 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.958015 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:55.958331 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:55.959222 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:24:55.959257 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:55.962261 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:55.965685 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:55.965695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:55.965712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:55.966330 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:55.966801 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:24:55.966805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:24:55.966808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:24:55.967016 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:24:55.967629 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:55.967635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:55.968214 (+ 579us) client_negotiation.cc:769] Sending connection context
0927 15:24:55.968547 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 15:24:55.969111 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":495,"threads_started":1}
I20260927 15:24:55.972539 21800 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42153
I20260927 15:24:55.972926 21800 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:55.973919 21800 heartbeater.cc:511] Master 127.2.246.126:42153 requested a full tablet report, sending...
I20260927 15:24:55.975809 21343 ts_manager.cc:194] Registered new tserver with Master: 9ff9e2fcdf9745758ae1ef19a574ff14 (127.2.246.67:37857)
I20260927 15:24:55.976737 21343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:49531
I20260927 15:24:55.984112 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:24:56.006507 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:55.989270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.2.246.126:42153)
0927 15:24:55.989620 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:55.989628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:55.990065 (+ 437us) server_negotiation.cc:407] Connection header received
0927 15:24:55.990315 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:55.990319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:55.990398 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:55.990514 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:55.992822 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.993650 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.995250 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:55.995745 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:55.995910 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:55.999512 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:55.999519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:55.999522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:55.999583 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:56.002878 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.004036 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.004040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.004043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.004213 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.004725 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.004729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.004731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.005118 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:56.005452 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.005883 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.006157 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 15:24:56.011924 21343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:24:56.021207 21343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:24:56.054028 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.037826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41919 (local address 127.0.0.1:55458)
0927 15:24:56.038710 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.038766 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.038965 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.040776 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.040791 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.040844 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.041582 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.041601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.044937 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.044948 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.049037 (+ 4089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.049081 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.052390 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.052403 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.052780 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.052875 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.053036 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.053371 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":380,"threads_started":1}
I20260927 15:24:56.056133 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.038172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.2.246.66:41919)
0927 15:24:56.039703 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.039718 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.039757 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:24:56.039894 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.039905 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.040339 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.041966 (+ 1627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.042296 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.044647 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.049559 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.051944 (+ 2385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.055184 (+ 3240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.055464 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.055557 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.055748 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1059,"threads_started":1}
I20260927 15:24:56.063081 21601 tablet_service.cc:1467] Processing CreateTablet for tablet da2c2addbc7845b39189258fda20e43c (DEFAULT_TABLE table=test-table [id=5b4169a512b241dda7805322b5bcddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:56.064368 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.039433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37857 (local address 127.0.0.1:55194)
0927 15:24:56.054806 (+ 15373us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.054834 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.054975 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.056268 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.056277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.056315 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.056520 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.056533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.058969 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.058975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.061021 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.061044 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.063473 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.063481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.063686 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.063749 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.063867 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.064054 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15233,"thread_start_us":16163,"threads_started":1}
I20260927 15:24:56.064626 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.039935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.2.246.67:37857)
0927 15:24:56.055247 (+ 15312us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.055257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.055295 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:24:56.055407 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.055416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.055726 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.056757 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.057088 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.058759 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.061286 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.063234 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.063427 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.063722 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.063962 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.064191 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15110,"thread_start_us":15985,"threads_started":1}
I20260927 15:24:56.065701 21601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2c2addbc7845b39189258fda20e43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:56.066522 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.040455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:35195 (local address 127.0.0.1:51552)
0927 15:24:56.055960 (+ 15505us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.055987 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.056194 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.058646 (+ 2452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.058651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.058686 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.059172 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.059200 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.060680 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.060685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.062693 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.062710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.065599 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.065609 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.065799 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.065875 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.065967 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.066115 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15340,"thread_start_us":16284,"threads_started":1}
I20260927 15:24:56.067363 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.040937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.2.246.65:35195)
0927 15:24:56.057487 (+ 16550us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.057497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.057527 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:24:56.057651 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.057659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.057963 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.058402 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.059387 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.060500 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.062889 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.064348 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.064970 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.065272 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.066844 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.067047 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16323,"thread_start_us":16413,"threads_started":1}
I20260927 15:24:56.071285 21467 tablet_service.cc:1467] Processing CreateTablet for tablet da2c2addbc7845b39189258fda20e43c (DEFAULT_TABLE table=test-table [id=5b4169a512b241dda7805322b5bcddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:56.070571 21735 tablet_service.cc:1467] Processing CreateTablet for tablet da2c2addbc7845b39189258fda20e43c (DEFAULT_TABLE table=test-table [id=5b4169a512b241dda7805322b5bcddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:56.072997 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2c2addbc7845b39189258fda20e43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:56.073431 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2c2addbc7845b39189258fda20e43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:56.086391 21819 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Bootstrap starting.
I20260927 15:24:56.089094 21820 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Bootstrap starting.
I20260927 15:24:56.091081 21819 tablet_bootstrap.cc:654] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:56.092530 21819 log.cc:824] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:56.095959 21819 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: No bootstrap required, opened a new log
I20260927 15:24:56.095981 21820 tablet_bootstrap.cc:654] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:56.096294 21819 ts_tablet_manager.cc:1397] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 15:24:56.097400 21820 log.cc:824] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:56.098003 21821 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Bootstrap starting.
I20260927 15:24:56.101487 21820 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: No bootstrap required, opened a new log
I20260927 15:24:56.102015 21820 ts_tablet_manager.cc:1397] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260927 15:24:56.102067 21821 tablet_bootstrap.cc:654] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:56.103570 21821 log.cc:824] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:56.106606 21819 raft_consensus.cc:348] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.107133 21819 raft_consensus.cc:374] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:56.107314 21819 raft_consensus.cc:729] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4751a29c4145fd8ff01629d3aacf7c, State: Initialized, Role: FOLLOWER
I20260927 15:24:56.107918 21821 tablet_bootstrap.cc:492] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: No bootstrap required, opened a new log
I20260927 15:24:56.108244 21821 ts_tablet_manager.cc:1397] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20260927 15:24:56.108052 21819 consensus_queue.cc:260] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.109671 21666 heartbeater.cc:503] Master 127.2.246.126:42153 was elected leader, sending a full tablet report...
I20260927 15:24:56.112013 21819 ts_tablet_manager.cc:1428] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260927 15:24:56.112968 21820 raft_consensus.cc:348] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.114259 21820 raft_consensus.cc:374] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:56.114461 21820 raft_consensus.cc:729] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928d0e1d81784dc6ac4b00c7a91a124f, State: Initialized, Role: FOLLOWER
I20260927 15:24:56.115424 21820 consensus_queue.cc:260] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.118021 21820 ts_tablet_manager.cc:1428] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 15:24:56.118571 21821 raft_consensus.cc:348] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.119120 21821 raft_consensus.cc:374] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:56.119238 21821 raft_consensus.cc:729] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff9e2fcdf9745758ae1ef19a574ff14, State: Initialized, Role: FOLLOWER
I20260927 15:24:56.119820 21821 consensus_queue.cc:260] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.121224 21800 heartbeater.cc:503] Master 127.2.246.126:42153 was elected leader, sending a full tablet report...
I20260927 15:24:56.121907 21821 ts_tablet_manager.cc:1428] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Time spent starting tablet: real 0.014s user 0.009s sys 0.007s
W20260927 15:24:56.169682 21533 tablet.cc:2369] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:24:56.195277 21801 tablet.cc:2369] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:56.224295 21826 raft_consensus.cc:482] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:56.224602 21826 raft_consensus.cc:504] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.226423 21826 leader_election.cc:288] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d4751a29c4145fd8ff01629d3aacf7c (127.2.246.66:41919), 9ff9e2fcdf9745758ae1ef19a574ff14 (127.2.246.67:37857)
I20260927 15:24:56.234807 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.227311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:58427 (local address 127.2.246.66:41919)
0927 15:24:56.227589 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.227595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.228443 (+ 848us) server_negotiation.cc:407] Connection header received
0927 15:24:56.228640 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.228646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.228815 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.229000 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.229817 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.230604 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.232638 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.233453 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.233665 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.233772 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.234334 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.234510 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:24:56.235015 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.227266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:41919 (local address 127.2.246.65:58427)
0927 15:24:56.228192 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.228219 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.228432 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.229054 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.229062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.229098 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.229646 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.229657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.230778 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.230786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.232449 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.232465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.233711 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.233717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.233968 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.233997 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.234302 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.234511 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":1,"spinlock_wait_cycles":22272,"thread_start_us":468,"threads_started":1}
I20260927 15:24:56.235864 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.228627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:49191 (local address 127.2.246.67:37857)
0927 15:24:56.228926 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.228935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.229322 (+ 387us) server_negotiation.cc:407] Connection header received
0927 15:24:56.229601 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.229607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.229694 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.229813 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.230564 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.231661 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.233649 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.234634 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.234865 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.234982 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.235464 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.235619 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:24:56.236321 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" is_pre_election: true
I20260927 15:24:56.236636 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.228355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37857 (local address 127.2.246.65:49191)
0927 15:24:56.229183 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.229213 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.229419 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.229905 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.229911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.229940 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.230369 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.230379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.231857 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.231870 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.233467 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.233481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.234917 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.234924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.235205 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.235253 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.236111 (+ 858us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.236310 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":581,"threads_started":1}
I20260927 15:24:56.236950 21621 raft_consensus.cc:2454] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928d0e1d81784dc6ac4b00c7a91a124f in term 0.
I20260927 15:24:56.237185 21825 raft_consensus.cc:482] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:56.237440 21825 raft_consensus.cc:504] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.237954 21423 leader_election.cc:302] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4751a29c4145fd8ff01629d3aacf7c, 928d0e1d81784dc6ac4b00c7a91a124f; no voters:
I20260927 15:24:56.238068 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" is_pre_election: true
I20260927 15:24:56.238610 21826 raft_consensus.cc:2781] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:56.238785 21826 raft_consensus.cc:482] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:56.238919 21826 raft_consensus.cc:3037] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:56.238782 21755 raft_consensus.cc:2454] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928d0e1d81784dc6ac4b00c7a91a124f in term 0.
I20260927 15:24:56.239579 21825 leader_election.cc:288] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ff9e2fcdf9745758ae1ef19a574ff14 (127.2.246.67:37857), 928d0e1d81784dc6ac4b00c7a91a124f (127.2.246.65:35195)
I20260927 15:24:56.243136 21826 raft_consensus.cc:504] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.244083 21826 leader_election.cc:288] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [CANDIDATE]: Term 1 election: Requested vote from peers 0d4751a29c4145fd8ff01629d3aacf7c (127.2.246.66:41919), 9ff9e2fcdf9745758ae1ef19a574ff14 (127.2.246.67:37857)
I20260927 15:24:56.244788 21621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4751a29c4145fd8ff01629d3aacf7c"
I20260927 15:24:56.245067 21621 raft_consensus.cc:3037] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:56.246071 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14"
I20260927 15:24:56.246331 21755 raft_consensus.cc:3037] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:56.250204 21755 raft_consensus.cc:2454] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928d0e1d81784dc6ac4b00c7a91a124f in term 1.
I20260927 15:24:56.250664 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.240644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:41027 (local address 127.2.246.67:37857)
0927 15:24:56.240839 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.240844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.241612 (+ 768us) server_negotiation.cc:407] Connection header received
0927 15:24:56.241826 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.241873 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.241984 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.242133 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.242944 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.244136 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.247503 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.249054 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.249241 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.249344 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.250127 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.250307 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:24:56.251333 21423 leader_election.cc:302] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 928d0e1d81784dc6ac4b00c7a91a124f, 9ff9e2fcdf9745758ae1ef19a574ff14; no voters:
I20260927 15:24:56.251698 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.240033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37857 (local address 127.2.246.66:41027)
0927 15:24:56.241465 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.241496 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.241656 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.242169 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.242177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.242219 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.242745 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.242762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.244369 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.244380 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.247276 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.247292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.249643 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.249651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.249887 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.249927 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.251001 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.251207 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":26,"thread_start_us":1040,"threads_started":1}
I20260927 15:24:56.251811 21826 raft_consensus.cc:2781] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:56.252446 21826 raft_consensus.cc:686] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 1 LEADER]: Becoming Leader. State: Replica: 928d0e1d81784dc6ac4b00c7a91a124f, State: Running, Role: LEADER
W20260927 15:24:56.253468 21667 tablet.cc:2369] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:56.253707 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.240824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:41127 (local address 127.2.246.65:35195)
0927 15:24:56.241223 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.241229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.245172 (+ 3943us) server_negotiation.cc:407] Connection header received
0927 15:24:56.245406 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.245412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.245504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.245615 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:56.246750 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.247869 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.250852 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.252343 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.252510 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.252605 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.253231 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.253395 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:24:56.253640 21826 consensus_queue.cc:237] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } }
I20260927 15:24:56.254576 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.240058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:35195 (local address 127.2.246.66:41127)
0927 15:24:56.245023 (+ 4965us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:56.245046 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:56.245229 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:56.245667 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:56.245674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:56.245710 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:56.246492 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.246507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.248071 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.248077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.250562 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:56.250576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.252807 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.252812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:56.252998 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.253036 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:24:56.253917 (+ 881us) client_negotiation.cc:240] Negotiation successful
0927 15:24:56.254093 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4783,"mutex_wait_us":5,"thread_start_us":10308,"threads_started":1}
I20260927 15:24:56.255916 21621 raft_consensus.cc:2454] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928d0e1d81784dc6ac4b00c7a91a124f in term 1.
I20260927 15:24:56.255877 21487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" is_pre_election: true
I20260927 15:24:56.257436 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2c2addbc7845b39189258fda20e43c" candidate_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" is_pre_election: true
I20260927 15:24:56.257774 21755 raft_consensus.cc:2379] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d4751a29c4145fd8ff01629d3aacf7c in current term 1: Already voted for candidate 928d0e1d81784dc6ac4b00c7a91a124f in this term.
I20260927 15:24:56.259159 21557 leader_election.cc:302] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d4751a29c4145fd8ff01629d3aacf7c; no voters: 928d0e1d81784dc6ac4b00c7a91a124f, 9ff9e2fcdf9745758ae1ef19a574ff14
I20260927 15:24:56.259784 21825 raft_consensus.cc:2726] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:24:56.265957 21342 catalog_manager.cc:5495] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f reported cstate change: term changed from 0 to 1, leader changed from <none> to 928d0e1d81784dc6ac4b00c7a91a124f (127.2.246.65). New cstate: current_term: 1 leader_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "928d0e1d81784dc6ac4b00c7a91a124f" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 35195 } health_report { overall_health: HEALTHY } } }
I20260927 15:24:56.322907 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.315806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.2.246.65:35195)
0927 15:24:56.316085 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.316090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.316117 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:24:56.316260 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.316264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.316367 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.316584 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:56.317182 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.318184 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.319809 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.320306 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.320431 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.321018 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:56.321778 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:24:56.322034 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.322272 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.322358 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 15:24:56.332520 21621 raft_consensus.cc:1261] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 1 FOLLOWER]: Refusing update from remote peer 928d0e1d81784dc6ac4b00c7a91a124f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:56.332492 21755 raft_consensus.cc:1261] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 1 FOLLOWER]: Refusing update from remote peer 928d0e1d81784dc6ac4b00c7a91a124f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:56.333698 21832 consensus_queue.cc:1035] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4751a29c4145fd8ff01629d3aacf7c" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:56.334348 21826 consensus_queue.cc:1035] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff9e2fcdf9745758ae1ef19a574ff14" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:56.349910 21835 mvcc.cc:205] Tried to move back new op lower bound from 7333980964165783552 to 7333980963863871488. Current Snapshot: MvccSnapshot[applied={T|T < 7333980964165783552}]
I20260927 15:24:56.372668 21342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:56.373100 21342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:56.378419 21342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fdd23bcb8f84c3ba8377f17ab754abe: Sending DeleteTablet for 3 replicas of tablet da2c2addbc7845b39189258fda20e43c
I20260927 15:24:56.380235 21601 tablet_service.cc:1514] Processing DeleteTablet for tablet da2c2addbc7845b39189258fda20e43c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:56 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:55458
I20260927 15:24:56.380489 21467 tablet_service.cc:1514] Processing DeleteTablet for tablet da2c2addbc7845b39189258fda20e43c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:56 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:51552
I20260927 15:24:56.380558 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet da2c2addbc7845b39189258fda20e43c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:56 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:55194
I20260927 15:24:56.382822 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21404
I20260927 15:24:56.391420 21844 tablet_replica.cc:330] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: stopping tablet replica
I20260927 15:24:56.392328 21846 tablet_replica.cc:330] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: stopping tablet replica
I20260927 15:24:56.393306 21846 raft_consensus.cc:2229] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:24:56.394147 21844 raft_consensus.cc:2229] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:56.395040 21844 raft_consensus.cc:2258] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:56.396174 21846 raft_consensus.cc:2258] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:56.397751 21845 tablet_replica.cc:330] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: stopping tablet replica
I20260927 15:24:56.398358 21845 raft_consensus.cc:2229] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:56.398962 21845 raft_consensus.cc:2258] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:56.400624 21844 ts_tablet_manager.cc:1905] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:56.401069 21846 ts_tablet_manager.cc:1905] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:56.402508 21845 ts_tablet_manager.cc:1905] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:56.409685 21844 ts_tablet_manager.cc:1918] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:56.409689 21846 ts_tablet_manager.cc:1918] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:56.409952 21844 log.cc:1196] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/da2c2addbc7845b39189258fda20e43c
I20260927 15:24:56.410060 21845 ts_tablet_manager.cc:1918] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:56.410300 21845 log.cc:1196] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/da2c2addbc7845b39189258fda20e43c
I20260927 15:24:56.410287 21846 log.cc:1196] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/da2c2addbc7845b39189258fda20e43c
I20260927 15:24:56.410652 21844 ts_tablet_manager.cc:1939] T da2c2addbc7845b39189258fda20e43c P 0d4751a29c4145fd8ff01629d3aacf7c: Deleting consensus metadata
I20260927 15:24:56.410895 21845 ts_tablet_manager.cc:1939] T da2c2addbc7845b39189258fda20e43c P 9ff9e2fcdf9745758ae1ef19a574ff14: Deleting consensus metadata
I20260927 15:24:56.411118 21846 ts_tablet_manager.cc:1939] T da2c2addbc7845b39189258fda20e43c P 928d0e1d81784dc6ac4b00c7a91a124f: Deleting consensus metadata
I20260927 15:24:56.413069 21330 catalog_manager.cc:4841] TS 9ff9e2fcdf9745758ae1ef19a574ff14 (127.2.246.67:37857): tablet da2c2addbc7845b39189258fda20e43c (table test-table [id=5b4169a512b241dda7805322b5bcddc0]) successfully deleted
I20260927 15:24:56.413213 21329 catalog_manager.cc:4841] TS 928d0e1d81784dc6ac4b00c7a91a124f (127.2.246.65:35195): tablet da2c2addbc7845b39189258fda20e43c (table test-table [id=5b4169a512b241dda7805322b5bcddc0]) successfully deleted
I20260927 15:24:56.413496 21330 catalog_manager.cc:4841] TS 0d4751a29c4145fd8ff01629d3aacf7c (127.2.246.66:41919): tablet da2c2addbc7845b39189258fda20e43c (table test-table [id=5b4169a512b241dda7805322b5bcddc0]) successfully deleted
Sep 27 15:24:56 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:24:56.426414 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.385899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.2.246.65:35195)
0927 15:24:56.386241 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.386252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.392791 (+ 6539us) server_negotiation.cc:407] Connection header received
0927 15:24:56.393113 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.393127 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.393255 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.393346 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:56.395431 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.396691 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.398873 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.400136 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.400306 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.417767 (+ 17461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:56.417787 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:56.417813 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:56.417939 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:56.421797 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.423163 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.423174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.423177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.423379 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.424025 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.424034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.424038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.424735 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:56.425158 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.425569 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.426024 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 15:24:56.427700 21527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:56.535494 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21404
I20260927 15:24:56.559548 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21538
Sep 27 15:24:56 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:24:56.578085 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.560539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.2.246.66:41919)
0927 15:24:56.560779 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.560786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.560912 (+ 126us) server_negotiation.cc:407] Connection header received
0927 15:24:56.561154 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.561158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.561241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.561362 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:56.562788 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.563549 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.564983 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.565550 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.565709 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.570726 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:56.570747 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:56.570769 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:56.570819 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:56.574500 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.575377 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.575385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.575388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.575563 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.576036 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.576040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.576043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.576554 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:56.576837 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.577201 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.577418 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:24:56.579107 21661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:56.682530 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21538
I20260927 15:24:56.701709 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21672
Sep 27 15:24:56 dist-test-slave-14jq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:24:56.720376 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:56.702811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.2.246.67:37857)
0927 15:24:56.703045 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:56.703051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:56.703074 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:24:56.703152 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:56.703157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:56.703236 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:56.703342 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:56.704949 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.705780 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.707381 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:56.707899 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:56.708055 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:56.713518 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:56.713548 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:56.713569 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:24:56.713631 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:56.716965 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.717986 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.717998 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.718002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.718174 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:24:56.718591 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:24:56.718598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:24:56.718602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:24:56.719039 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:56.719400 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:56.719704 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 15:24:56.719965 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:24:56.721253 21795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:56.826623 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21672
I20260927 15:24:56.845641 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21311
I20260927 15:24:56.847163 21373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:56.966797 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21311
2026-09-27T15:24:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:24:57Z Disabled control of system clock
I20260927 15:24:57.055595 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44273
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:24:57.419472 21855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:57.419809 21855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:57.419896 21855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:57.427495 21855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:24:57.427630 21855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:57.427668 21855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:57.427732 21855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:24:57.427770 21855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:24:57.437590 21855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:57.439060 21855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:57.440977 21855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:57.454453 21863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:57.454391 21861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:57.454478 21860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:57.455502 21855 server_base.cc:1034] running on GCE node
I20260927 15:24:57.456589 21855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:57.458580 21855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:57.459825 21855 hybrid_clock.cc:648] HybridClock initialized: now 1790522697459771 us; error 47 us; skew 500 ppm
I20260927 15:24:57.460263 21855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:57.463321 21855 webserver.cc:460] Webserver started at http://127.2.246.126:44729/ using document root <none> and password file <none>
I20260927 15:24:57.464241 21855 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:57.464344 21855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:57.464677 21855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:57.469730 21855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "bc34a0cf90e244018c0cd9d396f91a1a"
format_stamp: "Formatted at 2026-09-27 15:24:57 on dist-test-slave-14jq"
server_key: "20ff854195bd27752fcd63d9a4dcf3d9"
server_key_iv: "e0b6ff1f5d440739bbcb5ba7b52b80b3"
server_key_version: "encryptionkey@0"
I20260927 15:24:57.471197 21855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "bc34a0cf90e244018c0cd9d396f91a1a"
format_stamp: "Formatted at 2026-09-27 15:24:57 on dist-test-slave-14jq"
server_key: "20ff854195bd27752fcd63d9a4dcf3d9"
server_key_iv: "e0b6ff1f5d440739bbcb5ba7b52b80b3"
server_key_version: "encryptionkey@0"
I20260927 15:24:57.478964 21855 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 15:24:57.484014 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:57.486268 21855 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:24:57.486569 21855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "bc34a0cf90e244018c0cd9d396f91a1a"
format_stamp: "Formatted at 2026-09-27 15:24:57 on dist-test-slave-14jq"
server_key: "20ff854195bd27752fcd63d9a4dcf3d9"
server_key_iv: "e0b6ff1f5d440739bbcb5ba7b52b80b3"
server_key_version: "encryptionkey@0"
I20260927 15:24:57.486882 21855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:57.511184 21855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:57.512506 21855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:57.512907 21855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:57.538800 21855 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44273
I20260927 15:24:57.538828 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44273 every 8 connection(s)
I20260927 15:24:57.540704 21855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:24:57.543633 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21855
I20260927 15:24:57.543859 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:24:57.544518 3033 external_mini_cluster.cc:1428] Setting key 0ad5af6bbf970d5f05e749f38ef6d9f3
I20260927 15:24:57.546486 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:57.562294 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: Bootstrap starting.
I20260927 15:24:57.564208 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:57.547283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.2.246.126:44273)
0927 15:24:57.548718 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:57.548732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:57.548792 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:24:57.551660 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:57.551706 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:57.552583 (+ 877us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:57.553131 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:57.554155 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:57.555889 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:57.558026 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:57.559027 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:57.559247 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:57.560809 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:57.560823 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:57.560847 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:57.561629 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:57.561740 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:57.561778 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:57.561996 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:57.562329 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 15:24:57.562824 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":540,"threads_started":1}
I20260927 15:24:57.566228 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:57.567422 21921 log.cc:824] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:57.570878 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: No bootstrap required, opened a new log
I20260927 15:24:57.578866 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } }
I20260927 15:24:57.579425 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:57.579550 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc34a0cf90e244018c0cd9d396f91a1a, State: Initialized, Role: FOLLOWER
I20260927 15:24:57.580230 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } }
I20260927 15:24:57.580511 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:24:57.580670 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:24:57.580868 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:57.583278 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } }
I20260927 15:24:57.584084 21921 leader_election.cc:302] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc34a0cf90e244018c0cd9d396f91a1a; no voters:
I20260927 15:24:57.585083 21921 leader_election.cc:288] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:24:57.585299 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:57.588774 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [term 1 LEADER]: Becoming Leader. State: Replica: bc34a0cf90e244018c0cd9d396f91a1a, State: Running, Role: LEADER
I20260927 15:24:57.588992 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:24:57.589514 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } }
I20260927 15:24:57.594012 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } } }
I20260927 15:24:57.593950 21928 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc34a0cf90e244018c0cd9d396f91a1a. Latest consensus state: current_term: 1 leader_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc34a0cf90e244018c0cd9d396f91a1a" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44273 } } }
I20260927 15:24:57.594552 21928 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:57.594563 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a [sys.catalog]: This master's current role is: LEADER
I20260927 15:24:57.600557 21935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:24:57.608103 21935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:24:57.621785 21935 catalog_manager.cc:1347] Generated new cluster ID: e7e813a364dc4c46b800c6938691172a
I20260927 15:24:57.621968 21935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:24:57.647696 21935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:24:57.649389 21935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:24:57.660727 21935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: Generated new TSK 0
I20260927 15:24:57.661757 21935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:24:57.672935 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--builtin_ntp_servers=127.2.246.84:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:24:58.041819 21945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:58.042299 21945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:58.042455 21945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:58.050527 21945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:58.050681 21945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:58.050866 21945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:24:58.060770 21945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:58.062347 21945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:58.064442 21945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:58.080612 21951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:58.080700 21950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:58.080878 21953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:58.082331 21945 server_base.cc:1034] running on GCE node
I20260927 15:24:58.083105 21945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:58.084602 21945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:58.086130 21945 hybrid_clock.cc:648] HybridClock initialized: now 1790522698085993 us; error 93 us; skew 500 ppm
I20260927 15:24:58.086647 21945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:58.090446 21945 webserver.cc:460] Webserver started at http://127.2.246.65:37465/ using document root <none> and password file <none>
I20260927 15:24:58.091495 21945 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:58.091619 21945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:58.092367 21945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:58.097075 21945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "9782bf0fc98647c199f1d49f7e60858b"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "58525d0fc1973cd17feefa6ab9dd759e"
server_key_iv: "a68f19807cb2831ffddf601ce290b037"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.098453 21945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "9782bf0fc98647c199f1d49f7e60858b"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "58525d0fc1973cd17feefa6ab9dd759e"
server_key_iv: "a68f19807cb2831ffddf601ce290b037"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.106102 21945 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 15:24:58.111166 21959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.113298 21945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:24:58.113606 21945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "9782bf0fc98647c199f1d49f7e60858b"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "58525d0fc1973cd17feefa6ab9dd759e"
server_key_iv: "a68f19807cb2831ffddf601ce290b037"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.114003 21945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:58.161216 21945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:58.162712 21945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:58.163371 21945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:58.164825 21945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:58.167089 21945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:58.167246 21945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.167410 21945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:58.167536 21945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.213928 21945 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:42727
I20260927 15:24:58.214157 22071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:42727 every 8 connection(s)
I20260927 15:24:58.215922 21945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:24:58.222349 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 21945
I20260927 15:24:58.222571 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:24:58.223122 3033 external_mini_cluster.cc:1428] Setting key 72787725ebbd16fb55c4d04093f75fb4
I20260927 15:24:58.227012 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--builtin_ntp_servers=127.2.246.84:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:24:58.235641 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:58.219654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:49519 (local address 127.2.246.126:44273)
0927 15:24:58.220337 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:58.220345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:58.222029 (+ 1684us) server_negotiation.cc:407] Connection header received
0927 15:24:58.224852 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:58.224859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:58.224967 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:58.225123 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:58.227117 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.228563 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.230926 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.231755 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.231977 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:58.234353 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:58.234362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:58.234371 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:58.234405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:58.234472 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:58.234495 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:58.234612 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:58.234969 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 15:24:58.235112 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":365,"threads_started":1}
I20260927 15:24:58.237437 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:58.220337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44273 (local address 127.2.246.65:49519)
0927 15:24:58.221700 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:58.221754 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:58.224261 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:58.225541 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:58.225556 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:58.225749 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:58.226833 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:58.226860 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.228869 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.228876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:58.230660 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:58.230698 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.232199 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.232206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:58.232519 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:58.233584 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:58.233587 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:58.233653 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:58.234024 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:58.234654 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:58.234665 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:58.234674 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:24:58.234900 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 15:24:58.235378 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":418,"threads_started":1}
I20260927 15:24:58.239741 22072 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44273
I20260927 15:24:58.240435 22072 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:58.242290 22072 heartbeater.cc:511] Master 127.2.246.126:44273 requested a full tablet report, sending...
I20260927 15:24:58.245500 21886 ts_manager.cc:194] Registered new tserver with Master: 9782bf0fc98647c199f1d49f7e60858b (127.2.246.65:42727)
I20260927 15:24:58.247192 21886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.65:49519
W20260927 15:24:58.670815 22076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:58.671443 22076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:58.671655 22076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:58.680282 22076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:58.680495 22076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:58.680701 22076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:24:58.692006 22076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:58.694295 22076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:58.696727 22076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:58.715796 22081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:58.715878 22084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:58.716042 22082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:58.717067 22076 server_base.cc:1034] running on GCE node
I20260927 15:24:58.718185 22076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:58.719710 22076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:58.720992 22076 hybrid_clock.cc:648] HybridClock initialized: now 1790522698720917 us; error 64 us; skew 500 ppm
I20260927 15:24:58.721496 22076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:58.728314 22076 webserver.cc:460] Webserver started at http://127.2.246.66:33413/ using document root <none> and password file <none>
I20260927 15:24:58.729329 22076 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:58.729497 22076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:58.729986 22076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:58.733703 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "208d5568cd644822b1f12bf91d6809c3"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "1f5ac0bcc923dd9d8db464bc99ca61d2"
server_key_iv: "ab8cb76e472de3f444e3dd75ded8e073"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.734894 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "208d5568cd644822b1f12bf91d6809c3"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "1f5ac0bcc923dd9d8db464bc99ca61d2"
server_key_iv: "ab8cb76e472de3f444e3dd75ded8e073"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.744738 22076 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260927 15:24:58.750530 22090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.753093 22076 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:24:58.753444 22076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "208d5568cd644822b1f12bf91d6809c3"
format_stamp: "Formatted at 2026-09-27 15:24:58 on dist-test-slave-14jq"
server_key: "1f5ac0bcc923dd9d8db464bc99ca61d2"
server_key_iv: "ab8cb76e472de3f444e3dd75ded8e073"
server_key_version: "encryptionkey@0"
I20260927 15:24:58.753790 22076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:58.782445 22076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:58.783619 22076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:58.784013 22076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:58.785312 22076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:58.787251 22076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:58.787401 22076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.787552 22076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:58.787647 22076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:58.831364 22076 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:38243
I20260927 15:24:58.831458 22202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:38243 every 8 connection(s)
I20260927 15:24:58.833117 22076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:24:58.837376 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22076
I20260927 15:24:58.837577 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:24:58.838205 3033 external_mini_cluster.cc:1428] Setting key 3570ea96e309f7b7a79e4e96b3e04bf8
I20260927 15:24:58.841887 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--builtin_ntp_servers=127.2.246.84:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:24:58.851770 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:58.836648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:52975 (local address 127.2.246.126:44273)
0927 15:24:58.837241 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:58.837249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:58.839324 (+ 2075us) server_negotiation.cc:407] Connection header received
0927 15:24:58.841754 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:58.841759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:58.841911 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:58.842105 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:58.843863 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.844639 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.847388 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.847920 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.848091 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:58.850372 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:58.850378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:58.850383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:58.850414 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:58.850477 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:58.850494 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:58.850615 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:58.851081 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 15:24:58.851276 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":300,"threads_started":1}
I20260927 15:24:58.853122 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:58.837503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44273 (local address 127.2.246.66:52975)
0927 15:24:58.839043 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:58.839100 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:58.841395 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:58.842407 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:58.842424 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:58.842573 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:58.843488 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:58.843518 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.844830 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.844839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:58.847073 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:58.847095 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:58.848278 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:58.848288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:58.848674 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:58.849740 (+ 1066us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:58.849746 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:58.849820 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:58.849999 (+ 179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:58.850650 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:58.850665 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:58.850677 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:24:58.850910 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 15:24:58.851475 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":444,"threads_started":1}
I20260927 15:24:58.855122 22203 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44273
I20260927 15:24:58.855639 22203 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:58.857124 22203 heartbeater.cc:511] Master 127.2.246.126:44273 requested a full tablet report, sending...
I20260927 15:24:58.859709 21886 ts_manager.cc:194] Registered new tserver with Master: 208d5568cd644822b1f12bf91d6809c3 (127.2.246.66:38243)
I20260927 15:24:58.861135 21886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.66:52975
I20260927 15:24:59.252538 22072 heartbeater.cc:503] Master 127.2.246.126:44273 was elected leader, sending a full tablet report...
W20260927 15:24:59.261826 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:24:59.262249 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:24:59.262406 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:24:59.270114 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:24:59.270244 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:24:59.270416 22207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:24:59.280247 22207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:44273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:24:59.281694 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:24:59.283623 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:24:59.298717 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:59.298759 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:24:59.298780 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:24:59.300901 22207 server_base.cc:1034] running on GCE node
I20260927 15:24:59.301885 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:24:59.303256 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:24:59.304544 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790522699304486 us; error 44 us; skew 500 ppm
I20260927 15:24:59.305020 22207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:24:59.308446 22207 webserver.cc:460] Webserver started at http://127.2.246.67:37117/ using document root <none> and password file <none>
I20260927 15:24:59.309625 22207 fs_manager.cc:359] Metadata directory not provided
I20260927 15:24:59.309753 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:24:59.310181 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:24:59.313910 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "e426a100bb5c464caaa0c408fdacd740"
format_stamp: "Formatted at 2026-09-27 15:24:59 on dist-test-slave-14jq"
server_key: "ec4247d02085692c120b0a1350e0f38d"
server_key_iv: "6e2a2a5af8c0227d492959f5631a52dd"
server_key_version: "encryptionkey@0"
I20260927 15:24:59.314883 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "e426a100bb5c464caaa0c408fdacd740"
format_stamp: "Formatted at 2026-09-27 15:24:59 on dist-test-slave-14jq"
server_key: "ec4247d02085692c120b0a1350e0f38d"
server_key_iv: "6e2a2a5af8c0227d492959f5631a52dd"
server_key_version: "encryptionkey@0"
I20260927 15:24:59.322003 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:24:59.326859 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:59.328882 22207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:24:59.329172 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "e426a100bb5c464caaa0c408fdacd740"
format_stamp: "Formatted at 2026-09-27 15:24:59 on dist-test-slave-14jq"
server_key: "ec4247d02085692c120b0a1350e0f38d"
server_key_iv: "6e2a2a5af8c0227d492959f5631a52dd"
server_key_version: "encryptionkey@0"
I20260927 15:24:59.329480 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:24:59.355751 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:24:59.357036 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:24:59.357445 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:24:59.358851 22207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:24:59.361261 22207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:24:59.361408 22207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:59.361562 22207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:24:59.361655 22207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:24:59.409907 22207 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:37417
I20260927 15:24:59.410056 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:37417 every 8 connection(s)
I20260927 15:24:59.411676 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:24:59.422125 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22207
I20260927 15:24:59.422314 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:24:59.422786 3033 external_mini_cluster.cc:1428] Setting key c6686dfa0aaf4306382120397acad9a7
I20260927 15:24:59.427866 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.415203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:51453 (local address 127.2.246.126:44273)
0927 15:24:59.415713 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.415718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.417080 (+ 1362us) server_negotiation.cc:407] Connection header received
0927 15:24:59.419268 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.419275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.419386 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.419540 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:59.420782 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.421565 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.424147 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.424664 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.424841 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.426726 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:59.426733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:59.426737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:59.426773 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:59.426824 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:59.426839 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:59.426950 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.427247 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.427396 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260927 15:24:59.429621 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.415749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:44273 (local address 127.2.246.67:51453)
0927 15:24:59.416836 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.416896 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.418941 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.419800 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.419810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.419936 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:24:59.420589 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.420607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.422026 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.422034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.423861 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.423889 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.425007 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.425018 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.425332 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.426255 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:24:59.426258 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:24:59.426318 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:24:59.426446 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:24:59.426951 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:24:59.426965 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:24:59.426978 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.427223 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.427830 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":350,"threads_started":1}
I20260927 15:24:59.431980 22334 heartbeater.cc:347] Connected to a master server at 127.2.246.126:44273
I20260927 15:24:59.432587 22334 heartbeater.cc:464] Registering TS with master...
I20260927 15:24:59.434383 22334 heartbeater.cc:511] Master 127.2.246.126:44273 requested a full tablet report, sending...
I20260927 15:24:59.436956 21886 ts_manager.cc:194] Registered new tserver with Master: e426a100bb5c464caaa0c408fdacd740 (127.2.246.67:37417)
I20260927 15:24:59.438666 21886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.67:51453
I20260927 15:24:59.440231 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:24:59.455379 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.447093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.2.246.126:44273)
0927 15:24:59.447518 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.447527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.447938 (+ 411us) server_negotiation.cc:407] Connection header received
0927 15:24:59.448282 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.448287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.448374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.448509 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:59.449106 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.449983 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.451556 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.452016 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.452172 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.454082 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:59.454089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:59.454093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:59.454118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:59.454165 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:59.454181 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:59.454525 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.454876 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.454983 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 15:24:59.461659 21885 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:24:59.469408 21885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:24:59.502877 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.485882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.2.246.67:37417)
0927 15:24:59.489694 (+ 3812us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.489789 (+ 95us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.489878 (+ 89us) server_negotiation.cc:407] Connection header received
0927 15:24:59.490065 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.490080 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.490899 (+ 819us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.491208 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:59.494617 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.496135 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.499569 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.501255 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.501433 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.501860 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.502205 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.502418 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3521,"thread_start_us":3434,"threads_started":1}
I20260927 15:24:59.503490 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.485522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37417 (local address 127.0.0.1:37244)
0927 15:24:59.487263 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.487317 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.487556 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.492936 (+ 5380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.492946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.492984 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:59.494254 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.494287 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.496315 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.496322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.499190 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.499207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.501522 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.501531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.501805 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.501914 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.502112 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.502374 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":121,"thread_start_us":1911,"threads_started":1}
I20260927 15:24:59.504375 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.488981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54806)
0927 15:24:59.493728 (+ 4747us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.493756 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.494143 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.495343 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.495348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.495380 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:59.495821 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.495835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.498184 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.498191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.500709 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.500729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.503515 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.503525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.503743 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.503776 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.503870 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.504028 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4565,"thread_start_us":4376,"threads_started":1}
I20260927 15:24:59.505266 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.489391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.2.246.65:42727)
0927 15:24:59.492948 (+ 3557us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.492978 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.493932 (+ 954us) server_negotiation.cc:407] Connection header received
0927 15:24:59.494455 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.494469 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.494793 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.495001 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:59.496080 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.497864 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.501177 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.502849 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.503003 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.503354 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.504607 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.504845 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3232,"thread_start_us":3186,"threads_started":1}
I20260927 15:24:59.505775 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.485723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:38243 (local address 127.0.0.1:55650)
0927 15:24:59.492248 (+ 6525us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.492274 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.492677 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.493928 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.493933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.493991 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:59.494679 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.494695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.496908 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.496915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.501566 (+ 4651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.501583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.504902 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.504909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.505099 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.505132 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.505232 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.505406 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6350,"mutex_wait_us":103,"thread_start_us":6098,"threads_started":1}
I20260927 15:24:59.506206 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.486356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.2.246.66:38243)
0927 15:24:59.489009 (+ 2653us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.489021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.492504 (+ 3483us) server_negotiation.cc:407] Connection header received
0927 15:24:59.492977 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.492988 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.493342 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.493567 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:59.494899 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.496730 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.501913 (+ 5183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.504303 (+ 2390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.504661 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.504944 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.505343 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.505560 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2445,"thread_start_us":2283,"threads_started":1}
I20260927 15:24:59.508814 22269 tablet_service.cc:1467] Processing CreateTablet for tablet 47d69e375f05446bb001f6cb055730f4 (DEFAULT_TABLE table=test-table [id=d8b4f9089313435086af91d2b94a84f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:59.510001 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 47d69e375f05446bb001f6cb055730f4 (DEFAULT_TABLE table=test-table [id=d8b4f9089313435086af91d2b94a84f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:59.510001 22138 tablet_service.cc:1467] Processing CreateTablet for tablet 47d69e375f05446bb001f6cb055730f4 (DEFAULT_TABLE table=test-table [id=d8b4f9089313435086af91d2b94a84f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:24:59.510730 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d69e375f05446bb001f6cb055730f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:59.511662 22007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d69e375f05446bb001f6cb055730f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:59.511695 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d69e375f05446bb001f6cb055730f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:24:59.525174 22353 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Bootstrap starting.
I20260927 15:24:59.528985 22353 tablet_bootstrap.cc:654] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:59.529201 22355 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Bootstrap starting.
I20260927 15:24:59.531167 22353 log.cc:824] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:59.533341 22355 tablet_bootstrap.cc:654] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:59.534912 22354 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Bootstrap starting.
I20260927 15:24:59.536586 22355 log.cc:824] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:59.537595 22353 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: No bootstrap required, opened a new log
I20260927 15:24:59.538015 22353 ts_tablet_manager.cc:1397] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260927 15:24:59.539487 22354 tablet_bootstrap.cc:654] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Neither blocks nor log segments found. Creating new log.
I20260927 15:24:59.540848 22354 log.cc:824] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:24:59.548004 22355 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: No bootstrap required, opened a new log
I20260927 15:24:59.548408 22355 ts_tablet_manager.cc:1397] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260927 15:24:59.551635 22353 raft_consensus.cc:348] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.552212 22353 raft_consensus.cc:374] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:59.552340 22353 raft_consensus.cc:729] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e426a100bb5c464caaa0c408fdacd740, State: Initialized, Role: FOLLOWER
I20260927 15:24:59.553238 22354 tablet_bootstrap.cc:492] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: No bootstrap required, opened a new log
I20260927 15:24:59.553251 22353 consensus_queue.cc:260] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.553966 22354 ts_tablet_manager.cc:1397] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260927 15:24:59.562048 22353 ts_tablet_manager.cc:1428] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Time spent starting tablet: real 0.024s user 0.015s sys 0.007s
I20260927 15:24:59.562819 22334 heartbeater.cc:503] Master 127.2.246.126:44273 was elected leader, sending a full tablet report...
I20260927 15:24:59.562491 22355 raft_consensus.cc:348] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.563019 22355 raft_consensus.cc:374] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:59.563143 22355 raft_consensus.cc:729] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208d5568cd644822b1f12bf91d6809c3, State: Initialized, Role: FOLLOWER
I20260927 15:24:59.563930 22355 consensus_queue.cc:260] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.565234 22203 heartbeater.cc:503] Master 127.2.246.126:44273 was elected leader, sending a full tablet report...
I20260927 15:24:59.566083 22355 ts_tablet_manager.cc:1428] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260927 15:24:59.569821 22354 raft_consensus.cc:348] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.570436 22354 raft_consensus.cc:374] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:24:59.570564 22354 raft_consensus.cc:729] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9782bf0fc98647c199f1d49f7e60858b, State: Initialized, Role: FOLLOWER
I20260927 15:24:59.571367 22354 consensus_queue.cc:260] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.573547 22354 ts_tablet_manager.cc:1428] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Time spent starting tablet: real 0.019s user 0.010s sys 0.004s
W20260927 15:24:59.586058 22204 tablet.cc:2369] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:59.654722 22360 raft_consensus.cc:482] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:24:59.655081 22360 raft_consensus.cc:504] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.656787 22360 leader_election.cc:288] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e426a100bb5c464caaa0c408fdacd740 (127.2.246.67:37417), 9782bf0fc98647c199f1d49f7e60858b (127.2.246.65:42727)
W20260927 15:24:59.663980 22335 tablet.cc:2369] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:59.667596 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.657352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:37417 (local address 127.2.246.66:36643)
0927 15:24:59.658409 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.658426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.658584 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.659148 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.659154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.659195 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:59.659906 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.659934 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.661652 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.661658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.664689 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.664709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.666706 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.666712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.666893 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.666927 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.667074 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.667231 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":266,"spinlock_wait_cycles":37888,"thread_start_us":578,"threads_started":1}
I20260927 15:24:59.667584 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.657430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36643 (local address 127.2.246.67:37417)
0927 15:24:59.657663 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.657670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.658561 (+ 891us) server_negotiation.cc:407] Connection header received
0927 15:24:59.658858 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.658863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.658951 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.659072 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:59.660185 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.661443 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.665028 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.666385 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.666642 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.666733 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.667136 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.667284 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:24:59.668063 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.657518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:35097 (local address 127.2.246.65:42727)
0927 15:24:59.657711 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.657716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.658546 (+ 830us) server_negotiation.cc:407] Connection header received
0927 15:24:59.658800 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.658808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.658935 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.659092 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:24:59.660190 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.661442 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.665126 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.666580 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.666785 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.666855 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.667428 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.667561 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:24:59.668566 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.657294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.2.246.66:35097)
0927 15:24:59.658351 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.658375 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:24:59.658584 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:24:59.659148 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:24:59.659154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:24:59.659193 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:24:59.659933 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.659943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.661658 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.661663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.664748 (+ 3085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:24:59.664764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.666998 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.667003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:24:59.667221 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.667262 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:24:59.668064 (+ 802us) client_negotiation.cc:240] Negotiation successful
0927 15:24:59.668257 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":240,"spinlock_wait_cycles":14208,"thread_start_us":781,"threads_started":1}
I20260927 15:24:59.668864 22289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d69e375f05446bb001f6cb055730f4" candidate_uuid: "208d5568cd644822b1f12bf91d6809c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e426a100bb5c464caaa0c408fdacd740" is_pre_election: true
I20260927 15:24:59.669468 22289 raft_consensus.cc:2454] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 208d5568cd644822b1f12bf91d6809c3 in term 0.
I20260927 15:24:59.669965 22027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d69e375f05446bb001f6cb055730f4" candidate_uuid: "208d5568cd644822b1f12bf91d6809c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9782bf0fc98647c199f1d49f7e60858b" is_pre_election: true
I20260927 15:24:59.670495 22027 raft_consensus.cc:2454] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 208d5568cd644822b1f12bf91d6809c3 in term 0.
I20260927 15:24:59.670487 22093 leader_election.cc:302] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208d5568cd644822b1f12bf91d6809c3, e426a100bb5c464caaa0c408fdacd740; no voters:
I20260927 15:24:59.671125 22360 raft_consensus.cc:2781] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:24:59.671303 22360 raft_consensus.cc:482] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:24:59.671425 22360 raft_consensus.cc:3037] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:59.673733 22360 raft_consensus.cc:504] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.674496 22360 leader_election.cc:288] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [CANDIDATE]: Term 1 election: Requested vote from peers e426a100bb5c464caaa0c408fdacd740 (127.2.246.67:37417), 9782bf0fc98647c199f1d49f7e60858b (127.2.246.65:42727)
I20260927 15:24:59.675357 22289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d69e375f05446bb001f6cb055730f4" candidate_uuid: "208d5568cd644822b1f12bf91d6809c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e426a100bb5c464caaa0c408fdacd740"
I20260927 15:24:59.675357 22027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d69e375f05446bb001f6cb055730f4" candidate_uuid: "208d5568cd644822b1f12bf91d6809c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9782bf0fc98647c199f1d49f7e60858b"
I20260927 15:24:59.675679 22289 raft_consensus.cc:3037] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:59.675678 22027 raft_consensus.cc:3037] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:24:59.679652 22027 raft_consensus.cc:2454] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 208d5568cd644822b1f12bf91d6809c3 in term 1.
I20260927 15:24:59.679652 22289 raft_consensus.cc:2454] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 208d5568cd644822b1f12bf91d6809c3 in term 1.
I20260927 15:24:59.680442 22092 leader_election.cc:302] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208d5568cd644822b1f12bf91d6809c3, 9782bf0fc98647c199f1d49f7e60858b; no voters:
I20260927 15:24:59.680953 22360 raft_consensus.cc:2781] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:24:59.681608 22360 raft_consensus.cc:686] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 1 LEADER]: Becoming Leader. State: Replica: 208d5568cd644822b1f12bf91d6809c3, State: Running, Role: LEADER
I20260927 15:24:59.682235 22360 consensus_queue.cc:237] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } }
I20260927 15:24:59.687353 21885 catalog_manager.cc:5495] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 208d5568cd644822b1f12bf91d6809c3 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "208d5568cd644822b1f12bf91d6809c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "208d5568cd644822b1f12bf91d6809c3" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 38243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 } health_report { overall_health: UNKNOWN } } }
W20260927 15:24:59.719687 22073 tablet.cc:2369] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:24:59.752175 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.745912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.2.246.66:38243)
0927 15:24:59.746178 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.746184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.746208 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:24:59.746403 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.746407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.746514 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.746665 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:24:59.747085 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.747893 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.749567 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.750093 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.750211 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.750862 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:24:59.751430 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:24:59.751564 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.751764 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.751845 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:24:59.762326 22289 raft_consensus.cc:1261] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 1 FOLLOWER]: Refusing update from remote peer 208d5568cd644822b1f12bf91d6809c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:59.762537 22027 raft_consensus.cc:1261] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 1 FOLLOWER]: Refusing update from remote peer 208d5568cd644822b1f12bf91d6809c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:24:59.763444 22364 consensus_queue.cc:1035] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e426a100bb5c464caaa0c408fdacd740" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:59.763837 22360 consensus_queue.cc:1035] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9782bf0fc98647c199f1d49f7e60858b" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:24:59.802012 21886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:59.802366 21886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:24:59.806936 21886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc34a0cf90e244018c0cd9d396f91a1a: Sending DeleteTablet for 3 replicas of tablet 47d69e375f05446bb001f6cb055730f4
I20260927 15:24:59.808557 22269 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d69e375f05446bb001f6cb055730f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:59 UTC) from {username='slave'} at 127.0.0.1:37244
I20260927 15:24:59.808604 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d69e375f05446bb001f6cb055730f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:59 UTC) from {username='slave'} at 127.0.0.1:54806
I20260927 15:24:59.809530 22138 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d69e375f05446bb001f6cb055730f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:24:59 UTC) from {username='slave'} at 127.0.0.1:55650
I20260927 15:24:59.812266 22377 tablet_replica.cc:330] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: stopping tablet replica
I20260927 15:24:59.812791 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21945
I20260927 15:24:59.812999 22377 raft_consensus.cc:2229] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:59.813710 22377 raft_consensus.cc:2258] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:59.822495 22378 tablet_replica.cc:330] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: stopping tablet replica
I20260927 15:24:59.823379 22378 raft_consensus.cc:2229] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:24:59.824751 22377 ts_tablet_manager.cc:1905] T 47d69e375f05446bb001f6cb055730f4 P 9782bf0fc98647c199f1d49f7e60858b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:59.825889 22378 raft_consensus.cc:2258] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:59.826290 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.814929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.2.246.65:42727)
0927 15:24:59.815139 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.815146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.815373 (+ 227us) server_negotiation.cc:407] Connection header received
0927 15:24:59.815699 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.815707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.815838 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.815929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:59.816650 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.818257 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.820966 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.821761 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.821890 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.823609 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:59.823625 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:59.823648 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:59.825119 (+ 1471us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:59.825202 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:59.825232 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:59.825334 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.825618 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.825800 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:24:59.827919 22376 tablet_replica.cc:330] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: stopping tablet replica
I20260927 15:24:59.828114 22067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:24:59.828490 22376 raft_consensus.cc:2229] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:24:59.829222 22376 raft_consensus.cc:2258] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:24:59.832268 22376 ts_tablet_manager.cc:1905] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:59.842720 22376 ts_tablet_manager.cc:1918] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:59.842998 22376 log.cc:1196] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/47d69e375f05446bb001f6cb055730f4
I20260927 15:24:59.843673 22376 ts_tablet_manager.cc:1939] T 47d69e375f05446bb001f6cb055730f4 P e426a100bb5c464caaa0c408fdacd740: Deleting consensus metadata
I20260927 15:24:59.844295 22378 ts_tablet_manager.cc:1905] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:24:59.846256 21872 catalog_manager.cc:4841] TS e426a100bb5c464caaa0c408fdacd740 (127.2.246.67:37417): tablet 47d69e375f05446bb001f6cb055730f4 (table test-table [id=d8b4f9089313435086af91d2b94a84f4]) successfully deleted
I20260927 15:24:59.854324 22378 ts_tablet_manager.cc:1918] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:24:59.854599 22378 log.cc:1196] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/47d69e375f05446bb001f6cb055730f4
I20260927 15:24:59.855252 22378 ts_tablet_manager.cc:1939] T 47d69e375f05446bb001f6cb055730f4 P 208d5568cd644822b1f12bf91d6809c3: Deleting consensus metadata
I20260927 15:24:59.857777 21873 catalog_manager.cc:4841] TS 208d5568cd644822b1f12bf91d6809c3 (127.2.246.66:38243): tablet 47d69e375f05446bb001f6cb055730f4 (table test-table [id=d8b4f9089313435086af91d2b94a84f4]) successfully deleted
I20260927 15:24:59.944831 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21945
I20260927 15:24:59.964293 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22076
W20260927 15:24:59.964250 21871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:42727 (error 108)
I20260927 15:24:59.966706 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.966147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54820)
0927 15:24:59.966401 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:24:59.966524 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 15:24:59.967003 21871 catalog_manager.cc:4584] TS 9782bf0fc98647c199f1d49f7e60858b (127.2.246.65:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47d69e375f05446bb001f6cb055730f4: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
I20260927 15:24:59.973973 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:24:59.965537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.2.246.66:38243)
0927 15:24:59.965750 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:24:59.965755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:24:59.965776 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:24:59.965955 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:24:59.965960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:24:59.966079 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:24:59.966201 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:24:59.966770 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.967777 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.970331 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:24:59.970782 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:24:59.970898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:24:59.972362 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:24:59.972370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:24:59.972391 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:24:59.973089 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:24:59.973146 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:24:59.973168 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:24:59.973259 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:24:59.973455 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 15:24:59.973545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:24:59.974757 22198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:00.021298 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.020422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54828)
0927 15:25:00.020851 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.021049 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260927 15:25:00.022823 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.022263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54832)
0927 15:25:00.022476 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.022620 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:25:00.083822 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.082989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54840)
0927 15:25:00.083335 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.083496 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 15:25:00.085176 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.084693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54852)
0927 15:25:00.084912 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.085023 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 15:25:00.112138 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22076
I20260927 15:25:00.132728 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22207
I20260927 15:25:00.141077 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.133707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.2.246.67:37417)
0927 15:25:00.134033 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:00.134039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:00.134070 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:25:00.134288 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:00.134293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:00.134383 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:00.134542 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:00.135193 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:00.135956 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:00.137553 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:00.138191 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:00.138332 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:00.139648 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:00.139658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:00.139673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:00.140326 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:00.140380 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:00.140400 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:00.140511 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:00.140675 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 15:25:00.140807 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 15:25:00.141956 22329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:00.166532 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.165760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54858)
0927 15:25:00.166146 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.166330 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 15:25:00.168131 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.167633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54874)
0927 15:25:00.167873 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.167993 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 15:25:00.247066 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22207
I20260927 15:25:00.265702 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 21855
I20260927 15:25:00.267444 21916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:00.388852 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:00.387690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:42727 (local address 127.0.0.1:54880)
0927 15:25:00.388423 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:00.388564 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550}
I20260927 15:25:00.389187 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 21855
2026-09-27T15:25:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3417 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:00 dist-test-slave-14jq krb5kdc[22387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:00 dist-test-slave-14jq krb5kdc[22387](info): set up 2 sockets
Sep 27 15:25:00 dist-test-slave-14jq krb5kdc[22387](info): commencing operation
krb5kdc: starting...
W20260927 15:25:01.542490 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.006s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:01 dist-test-slave-14jq krb5kdc[22387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:01.887693 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37375
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:02.273697 22400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:02.274326 22400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:02.274554 22400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:02.286154 22400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:02.286307 22400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:02.286383 22400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:02.286481 22400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:02.286549 22400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:02.297233 22400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:37375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:02.298775 22400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:02.300153 22400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:02.314327 22405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:02.314170 22406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:02.314216 22408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:02.317333 22400 server_base.cc:1034] running on GCE node
I20260927 15:25:02.318503 22400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:02.320382 22400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:02.321719 22400 hybrid_clock.cc:648] HybridClock initialized: now 1790522702321629 us; error 81 us; skew 500 ppm
I20260927 15:25:02.322259 22400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:02 dist-test-slave-14jq krb5kdc[22387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:02.326408 22400 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:02.328454 22400 webserver.cc:460] Webserver started at http://127.2.246.126:39973/ using document root <none> and password file <none>
I20260927 15:25:02.329255 22400 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:02.329428 22400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:02.329763 22400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:02.332986 22400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3"
format_stamp: "Formatted at 2026-09-27 15:25:02 on dist-test-slave-14jq"
server_key: "1b31be7d2d4666b5e3d129c38fb7a64d"
server_key_iv: "0f9b8228cf0fc7da7453d5d754d34f3c"
server_key_version: "encryptionkey@0"
I20260927 15:25:02.333812 22400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3"
format_stamp: "Formatted at 2026-09-27 15:25:02 on dist-test-slave-14jq"
server_key: "1b31be7d2d4666b5e3d129c38fb7a64d"
server_key_iv: "0f9b8228cf0fc7da7453d5d754d34f3c"
server_key_version: "encryptionkey@0"
I20260927 15:25:02.340453 22400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:25:02.345166 22415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:02.347652 22400 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 15:25:02.347994 22400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3"
format_stamp: "Formatted at 2026-09-27 15:25:02 on dist-test-slave-14jq"
server_key: "1b31be7d2d4666b5e3d129c38fb7a64d"
server_key_iv: "0f9b8228cf0fc7da7453d5d754d34f3c"
server_key_version: "encryptionkey@0"
I20260927 15:25:02.348285 22400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:02.375914 22400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:02.381762 22400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:02.382409 22400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:02.407184 22400 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:37375
I20260927 15:25:02.407364 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:37375 every 8 connection(s)
I20260927 15:25:02.408869 22400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:02.413727 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22400
I20260927 15:25:02.414019 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:02.414677 3033 external_mini_cluster.cc:1428] Setting key 311b9457076c4c9fc9fb03e9a59d8c67
I20260927 15:25:02.415010 22467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:25:02 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:02.437543 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: Bootstrap starting.
I20260927 15:25:02.442777 22467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:02.444216 22467 log.cc:824] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:02.447558 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:02.417178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.2.246.126:37375)
0927 15:25:02.418217 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:02.418230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:02.418281 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:25:02.421111 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:02.421143 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:02.421693 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:02.422250 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:02.425360 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:02.427284 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:02.430248 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:02.431271 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:02.431476 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:02.438156 (+ 6680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:02.438168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:02.438189 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:02.438253 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:02.442235 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:02.443315 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:02.443324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:02.443333 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:02.443537 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:02.443987 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:02.443992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:02.443996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:02.444683 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:02.445052 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:02.445534 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 15:25:02.446090 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":464,"threads_started":1}
I20260927 15:25:02.448491 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: No bootstrap required, opened a new log
I20260927 15:25:02.457762 22467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } }
I20260927 15:25:02.458338 22467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:02.458449 22467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677ba9abe99e4a4a9b5f4187cb15a0d3, State: Initialized, Role: FOLLOWER
I20260927 15:25:02.459064 22467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } }
I20260927 15:25:02.459282 22467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:02.459378 22467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:02.459512 22467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:02.461792 22467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } }
I20260927 15:25:02.462357 22467 leader_election.cc:302] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677ba9abe99e4a4a9b5f4187cb15a0d3; no voters:
I20260927 15:25:02.463137 22467 leader_election.cc:288] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:02.463434 22472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:02.464941 22472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [term 1 LEADER]: Becoming Leader. State: Replica: 677ba9abe99e4a4a9b5f4187cb15a0d3, State: Running, Role: LEADER
I20260927 15:25:02.465924 22467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:02.465803 22472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } }
I20260927 15:25:02.470309 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677ba9abe99e4a4a9b5f4187cb15a0d3. Latest consensus state: current_term: 1 leader_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } } }
I20260927 15:25:02.470929 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:02.470264 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677ba9abe99e4a4a9b5f4187cb15a0d3" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37375 } } }
I20260927 15:25:02.471436 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:02.473881 22482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:02.483112 22482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:02.495452 22482 catalog_manager.cc:1347] Generated new cluster ID: 7a69cb11504b4ce292c4223ec2b996f2
I20260927 15:25:02.495640 22482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:02.509756 22482 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:02.511511 22482 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:02.524953 22482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: Generated new TSK 0
I20260927 15:25:02.526058 22482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
Entry for principal kudu/127.2.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab.
I20260927 15:25:02.611233 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:02.985040 22493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:02.985366 22493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:02.985476 22493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:02.993780 22493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:02.993990 22493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:02.994117 22493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:25:03.004496 22493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.65.keytab
--principal=kudu/127.2.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:03.005965 22493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:03.007267 22493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:03.021777 22498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:03.021828 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:03.021965 22499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:03.024295 22493 server_base.cc:1034] running on GCE node
I20260927 15:25:03.025311 22493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:03.026759 22493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:03.028123 22493 hybrid_clock.cc:648] HybridClock initialized: now 1790522703028034 us; error 83 us; skew 500 ppm
I20260927 15:25:03.028666 22493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:03 dist-test-slave-14jq krb5kdc[22387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:03.034682 22493 init.cc:377] Logged in from keytab as kudu/127.2.246.65@KRBTEST.COM (short username kudu)
I20260927 15:25:03.037077 22493 webserver.cc:460] Webserver started at http://127.2.246.65:38285/ using document root <none> and password file <none>
I20260927 15:25:03.038242 22493 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:03.038439 22493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:03.038874 22493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:03.044360 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "0a2f69e2cef74a8cb97af3e5eefe9626"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "828467f0d7fe8b9a0578d9c4d42c4227"
server_key_iv: "59eb3be23505f8e0731196f216ab4585"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.045560 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "0a2f69e2cef74a8cb97af3e5eefe9626"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "828467f0d7fe8b9a0578d9c4d42c4227"
server_key_iv: "59eb3be23505f8e0731196f216ab4585"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.053516 22493 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 15:25:03.060343 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.062808 22493 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:25:03.063148 22493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "0a2f69e2cef74a8cb97af3e5eefe9626"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "828467f0d7fe8b9a0578d9c4d42c4227"
server_key_iv: "59eb3be23505f8e0731196f216ab4585"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.063457 22493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:03.080971 22493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:03.087005 22493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:03.087386 22493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:03.088696 22493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:03.090859 22493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:03.091017 22493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.091233 22493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:03.091311 22493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.137068 22493 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:45215
I20260927 15:25:03.137126 22620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:45215 every 8 connection(s)
I20260927 15:25:03.139029 22493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:25:03.142457 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22493
I20260927 15:25:03.142684 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:25:03.143208 3033 external_mini_cluster.cc:1428] Setting key a8ae4ddafdd4a1b02f52f3eefe06680d
Sep 27 15:25:03 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.65@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:03.167920 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:03.142370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:51013 (local address 127.2.246.126:37375)
0927 15:25:03.144410 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:03.144418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:03.144779 (+ 361us) server_negotiation.cc:407] Connection header received
0927 15:25:03.147498 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:03.147507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:03.147625 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:03.147785 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:03.150020 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.151210 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.154049 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.154860 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.155007 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:03.161224 (+ 6217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:03.161230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:03.161234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:03.161275 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:03.164155 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:03.165084 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:03.165091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:03.165100 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:03.165229 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:03.165696 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:03.165707 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:03.165709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:03.166110 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:03.166363 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:03.167224 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 15:25:03.167503 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1872,"thread_start_us":2228,"threads_started":1}
I20260927 15:25:03.169332 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:03.143030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37375 (local address 127.2.246.65:51013)
0927 15:25:03.144413 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:03.144539 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:03.147047 (+ 2508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:03.148190 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:03.148203 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:03.148909 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:03.149716 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:03.149739 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.151474 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.151480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:03.153680 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:03.153709 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.155590 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.155612 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:03.156136 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:03.157307 (+ 1171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:03.157358 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:03.160880 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:03.164371 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:03.164384 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:03.164412 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:03.164943 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:03.165348 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:03.165360 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:03.165364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:03.165571 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:03.166302 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:03.166312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:03.166818 (+ 506us) client_negotiation.cc:769] Sending connection context
0927 15:25:03.167134 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 15:25:03.167773 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":501,"threads_started":1}
I20260927 15:25:03.171368 22621 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37375
I20260927 15:25:03.171754 22621 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:03.172813 22621 heartbeater.cc:511] Master 127.2.246.126:37375 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.246.66@KRBTEST.COM; defaulting to no policy
I20260927 15:25:03.175715 22432 ts_manager.cc:194] Registered new tserver with Master: 0a2f69e2cef74a8cb97af3e5eefe9626 (127.2.246.65:45215)
I20260927 15:25:03.177798 22432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.65@KRBTEST.COM'} at 127.2.246.65:51013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
Entry for principal kudu/127.2.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab.
I20260927 15:25:03.219453 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:03.595360 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:03.595687 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:03.595795 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:03.604326 22627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:03.604476 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:03.604595 22627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:25:03.618597 22627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.66.keytab
--principal=kudu/127.2.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:03.620085 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:03.621397 22627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:03.637878 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:03.637878 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:03.637961 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:03.639747 22627 server_base.cc:1034] running on GCE node
I20260927 15:25:03.640527 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:03.641721 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:03.643018 22627 hybrid_clock.cc:648] HybridClock initialized: now 1790522703642970 us; error 43 us; skew 500 ppm
I20260927 15:25:03.643448 22627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:03 dist-test-slave-14jq krb5kdc[22387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:03.647748 22627 init.cc:377] Logged in from keytab as kudu/127.2.246.66@KRBTEST.COM (short username kudu)
I20260927 15:25:03.649770 22627 webserver.cc:460] Webserver started at http://127.2.246.66:39141/ using document root <none> and password file <none>
I20260927 15:25:03.650614 22627 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:03.650784 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:03.651144 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:03.654606 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "47f2b60c190449119e92c1428e039c49"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "03eaa4a3f12ecdfc90cb72a9e24f5666"
server_key_iv: "2160133792f31f2e79a10246781e9f50"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.655532 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "47f2b60c190449119e92c1428e039c49"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "03eaa4a3f12ecdfc90cb72a9e24f5666"
server_key_iv: "2160133792f31f2e79a10246781e9f50"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.661747 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:25:03.666306 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.669047 22627 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:25:03.669327 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "47f2b60c190449119e92c1428e039c49"
format_stamp: "Formatted at 2026-09-27 15:25:03 on dist-test-slave-14jq"
server_key: "03eaa4a3f12ecdfc90cb72a9e24f5666"
server_key_iv: "2160133792f31f2e79a10246781e9f50"
server_key_version: "encryptionkey@0"
I20260927 15:25:03.669703 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:03.688263 22627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:03.693370 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:03.693706 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:03.694952 22627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:03.696602 22627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:03.696720 22627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.696902 22627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:03.696982 22627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:03.741117 22627 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:43701
I20260927 15:25:03.741375 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:43701 every 8 connection(s)
I20260927 15:25:03.742660 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:25:03.752213 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22627
I20260927 15:25:03.752472 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:25:03.753173 3033 external_mini_cluster.cc:1428] Setting key 29c08e89db04e7d6bae15883c8657c4c
Sep 27 15:25:03 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.66@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:03.774807 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:03.746900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:36841 (local address 127.2.246.126:37375)
0927 15:25:03.747648 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:03.747653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:03.748472 (+ 819us) server_negotiation.cc:407] Connection header received
0927 15:25:03.750722 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:03.750728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:03.750827 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:03.750992 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:03.752917 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.754283 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.756217 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.756706 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.756868 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:03.765547 (+ 8679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:03.765554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:03.765558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:03.765606 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:03.769940 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:03.771388 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:03.771397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:03.771403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:03.771604 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:03.772279 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:03.772287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:03.772293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:03.772739 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:03.772909 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:03.773984 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 15:25:03.774315 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":314,"threads_started":1}
I20260927 15:25:03.776733 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:03.747246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37375 (local address 127.2.246.66:36841)
0927 15:25:03.748228 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:03.748280 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:03.750366 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:03.751240 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:03.751250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:03.751872 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:03.752642 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:03.752673 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.754522 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.754528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:03.756036 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:03.756051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:03.756974 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:03.756982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:03.757235 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:03.761642 (+ 4407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:03.761674 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:03.765055 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:03.770240 (+ 5185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:03.770253 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:03.770276 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:03.771171 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:03.771803 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:03.771807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:03.771809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:03.772118 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:03.772929 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:03.772938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:03.773469 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 15:25:03.773841 (+ 372us) client_negotiation.cc:240] Negotiation successful
0927 15:25:03.774592 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":417,"threads_started":1}
I20260927 15:25:03.779101 22755 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37375
I20260927 15:25:03.779546 22755 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:03.780684 22755 heartbeater.cc:511] Master 127.2.246.126:37375 requested a full tablet report, sending...
I20260927 15:25:03.783494 22432 ts_manager.cc:194] Registered new tserver with Master: 47f2b60c190449119e92c1428e039c49 (127.2.246.66:43701)
WARNING: no policy specified for kudu/127.2.246.67@KRBTEST.COM; defaulting to no policy
I20260927 15:25:03.784844 22432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.66@KRBTEST.COM'} at 127.2.246.66:36841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
Entry for principal kudu/127.2.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab.
I20260927 15:25:03.822103 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:04.176400 22761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:04.176774 22761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:04.176949 22761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:25:04.182097 22621 heartbeater.cc:503] Master 127.2.246.126:37375 was elected leader, sending a full tablet report...
W20260927 15:25:04.184903 22761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:04.185087 22761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:04.185286 22761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:25:04.196506 22761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.67.keytab
--principal=kudu/127.2.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:37375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:04.198072 22761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:04.199514 22761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:04.216367 22767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:04.216467 22766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:04.216332 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:04.219873 22761 server_base.cc:1034] running on GCE node
I20260927 15:25:04.220675 22761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:04.221985 22761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:04.223253 22761 hybrid_clock.cc:648] HybridClock initialized: now 1790522704223191 us; error 56 us; skew 500 ppm
I20260927 15:25:04.223712 22761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:04 dist-test-slave-14jq krb5kdc[22387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:04.227882 22761 init.cc:377] Logged in from keytab as kudu/127.2.246.67@KRBTEST.COM (short username kudu)
I20260927 15:25:04.230064 22761 webserver.cc:460] Webserver started at http://127.2.246.67:36915/ using document root <none> and password file <none>
I20260927 15:25:04.230928 22761 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:04.231103 22761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:04.231503 22761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:04.234920 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "3790ec5d5aa34650b987539520e6d120"
format_stamp: "Formatted at 2026-09-27 15:25:04 on dist-test-slave-14jq"
server_key: "e8171c4233fe2f345002cdc648b1ac91"
server_key_iv: "6a9d9823a54d240d8dc000a17fde47a8"
server_key_version: "encryptionkey@0"
I20260927 15:25:04.235821 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "3790ec5d5aa34650b987539520e6d120"
format_stamp: "Formatted at 2026-09-27 15:25:04 on dist-test-slave-14jq"
server_key: "e8171c4233fe2f345002cdc648b1ac91"
server_key_iv: "6a9d9823a54d240d8dc000a17fde47a8"
server_key_version: "encryptionkey@0"
I20260927 15:25:04.242067 22761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:25:04.246565 22776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:04.249037 22761 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:25:04.249301 22761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "3790ec5d5aa34650b987539520e6d120"
format_stamp: "Formatted at 2026-09-27 15:25:04 on dist-test-slave-14jq"
server_key: "e8171c4233fe2f345002cdc648b1ac91"
server_key_iv: "6a9d9823a54d240d8dc000a17fde47a8"
server_key_version: "encryptionkey@0"
I20260927 15:25:04.249573 22761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:04.268934 22761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:04.276531 22761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:04.276952 22761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:04.278744 22761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:04.281198 22761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:04.281340 22761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:04.281515 22761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:04.281586 22761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:04.326289 22761 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:39187
I20260927 15:25:04.326396 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:39187 every 8 connection(s)
I20260927 15:25:04.327805 22761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:25:04.336520 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22761
I20260927 15:25:04.336716 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:25:04.337316 3033 external_mini_cluster.cc:1428] Setting key c23d366819d4051e7a28e7ec629b86bb
Sep 27 15:25:04 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.67@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:04.356873 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.331401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:44581 (local address 127.2.246.126:37375)
0927 15:25:04.332010 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.332016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.333349 (+ 1333us) server_negotiation.cc:407] Connection header received
0927 15:25:04.335679 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.335686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.335810 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.335956 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:04.338663 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.339890 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.342428 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.342931 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.343047 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.348273 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:04.348279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:04.348284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:04.348332 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:04.351382 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.353058 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.353064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.353068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.353281 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.354233 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.354238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.354242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.354723 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:04.355019 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.356238 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.356521 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":291,"threads_started":1}
I20260927 15:25:04.358762 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.332022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:37375 (local address 127.2.246.67:44581)
0927 15:25:04.332932 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.333181 (+ 249us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.335314 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.336312 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.336323 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.337328 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:04.338414 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.338446 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.340225 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.340233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.342240 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.342259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.343182 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.343188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.343733 (+ 545us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.344706 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:04.344738 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:04.347867 (+ 3129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:04.351679 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:04.351694 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:04.351730 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:04.352840 (+ 1110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:04.353519 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:04.353525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:04.353530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:04.354037 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:04.355109 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:04.355117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:04.355671 (+ 554us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.356064 (+ 393us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.356983 (+ 919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":345,"threads_started":1}
I20260927 15:25:04.361259 22889 heartbeater.cc:347] Connected to a master server at 127.2.246.126:37375
I20260927 15:25:04.361788 22889 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:04.362972 22889 heartbeater.cc:511] Master 127.2.246.126:37375 requested a full tablet report, sending...
I20260927 15:25:04.365473 22432 ts_manager.cc:194] Registered new tserver with Master: 3790ec5d5aa34650b987539520e6d120 (127.2.246.67:39187)
I20260927 15:25:04.366909 22432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.246.67@KRBTEST.COM'} at 127.2.246.67:44581
I20260927 15:25:04.368570 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:25:04.391884 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.374752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.2.246.126:37375)
0927 15:25:04.375068 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.375076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.375385 (+ 309us) server_negotiation.cc:407] Connection header received
0927 15:25:04.375827 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.375832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.375926 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.376054 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:04.378202 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.379052 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.380697 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.381132 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.381329 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.384401 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:04.384407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:04.384411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:04.384457 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:04.387592 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.388993 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.388998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.389000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.389176 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.389975 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.389979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.389981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.390357 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:04.390620 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.391286 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.391506 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 15:25:04.397701 22432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:25:04.406495 22432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:25:04.448321 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.425569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.2.246.65:45215)
0927 15:25:04.428753 (+ 3184us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.428770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.428819 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:25:04.428992 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.429006 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.433581 (+ 4575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.434409 (+ 828us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:04.437535 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.439979 (+ 2444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.444204 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.446146 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.446323 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.446661 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.447592 (+ 931us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.447852 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":2621,"threads_started":1}
I20260927 15:25:04.449361 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.425079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45215 (local address 127.0.0.1:36750)
0927 15:25:04.426470 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.426527 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.426759 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.435448 (+ 8689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.435475 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.435561 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:04.436708 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.436751 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.440354 (+ 3603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.440365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.443954 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.443971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.446889 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.446897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.447230 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.447308 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.448394 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.448988 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":1,"thread_start_us":15734,"threads_started":1}
I20260927 15:25:04.451279 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.429633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:39187 (local address 127.0.0.1:48862)
0927 15:25:04.441538 (+ 11905us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.441567 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.441750 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.442851 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.442856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.442885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:04.443574 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.443590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.445407 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.445412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.447631 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.447646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.450160 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.450170 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.450365 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.450401 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.450505 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.450699 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11553,"mutex_wait_us":147,"thread_start_us":10924,"threads_started":1}
I20260927 15:25:04.451915 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.426186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.2.246.66:43701)
0927 15:25:04.433359 (+ 7173us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.433371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.441341 (+ 7970us) server_negotiation.cc:407] Connection header received
0927 15:25:04.441742 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.441753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.442091 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.442349 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:04.443476 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.445243 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.448512 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.450108 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.450259 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.450444 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.451392 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.451585 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6902,"thread_start_us":6797,"threads_started":1}
I20260927 15:25:04.452078 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.430145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.2.246.67:39187)
0927 15:25:04.439839 (+ 9694us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.439854 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.441939 (+ 2085us) server_negotiation.cc:407] Connection header received
0927 15:25:04.442091 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.442103 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.442604 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.442861 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:04.443912 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.445179 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.447917 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.449498 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.449660 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.449996 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.451548 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.451727 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9376,"thread_start_us":9100,"threads_started":1}
I20260927 15:25:04.452454 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.425437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43701 (local address 127.0.0.1:50124)
0927 15:25:04.441069 (+ 15632us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.441093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.441476 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.442603 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.442611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.442649 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:04.443218 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.443234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.445486 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.445492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.448186 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.448205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.450699 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.450705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.450946 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.450977 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.451072 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.451229 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15424,"mutex_wait_us":4,"thread_start_us":987,"threads_started":1}
I20260927 15:25:04.455261 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 (DEFAULT_TABLE table=test-table [id=e1dbb75640e14e73b6d418bdf9463df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:04.456225 22690 tablet_service.cc:1467] Processing CreateTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 (DEFAULT_TABLE table=test-table [id=e1dbb75640e14e73b6d418bdf9463df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:04.456579 22824 tablet_service.cc:1467] Processing CreateTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 (DEFAULT_TABLE table=test-table [id=e1dbb75640e14e73b6d418bdf9463df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:04.457351 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d46c69d585c4d5eb872d4c79fe09292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:04.457669 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d46c69d585c4d5eb872d4c79fe09292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:04.458096 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d46c69d585c4d5eb872d4c79fe09292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:04.469470 22908 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Bootstrap starting.
I20260927 15:25:04.473488 22908 tablet_bootstrap.cc:654] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:04.474843 22908 log.cc:824] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:04.478179 22908 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: No bootstrap required, opened a new log
I20260927 15:25:04.478152 22910 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Bootstrap starting.
I20260927 15:25:04.478487 22908 ts_tablet_manager.cc:1397] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 15:25:04.482874 22911 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Bootstrap starting.
I20260927 15:25:04.482975 22910 tablet_bootstrap.cc:654] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:04.484495 22910 log.cc:824] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:04.488276 22911 tablet_bootstrap.cc:654] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:04.488737 22910 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: No bootstrap required, opened a new log
I20260927 15:25:04.489137 22910 ts_tablet_manager.cc:1397] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260927 15:25:04.488843 22908 raft_consensus.cc:348] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.489338 22908 raft_consensus.cc:374] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:04.489440 22908 raft_consensus.cc:729] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f2b60c190449119e92c1428e039c49, State: Initialized, Role: FOLLOWER
I20260927 15:25:04.490108 22911 log.cc:824] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:04.490150 22908 consensus_queue.cc:260] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.491698 22755 heartbeater.cc:503] Master 127.2.246.126:37375 was elected leader, sending a full tablet report...
I20260927 15:25:04.493716 22908 ts_tablet_manager.cc:1428] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260927 15:25:04.495287 22756 tablet.cc:2369] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:04.497762 22911 tablet_bootstrap.cc:492] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: No bootstrap required, opened a new log
I20260927 15:25:04.498193 22911 ts_tablet_manager.cc:1397] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260927 15:25:04.500977 22910 raft_consensus.cc:348] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.501469 22910 raft_consensus.cc:374] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:04.501590 22910 raft_consensus.cc:729] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2f69e2cef74a8cb97af3e5eefe9626, State: Initialized, Role: FOLLOWER
I20260927 15:25:04.502267 22910 consensus_queue.cc:260] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.503870 22910 ts_tablet_manager.cc:1428] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260927 15:25:04.509718 22911 raft_consensus.cc:348] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.510391 22911 raft_consensus.cc:374] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:04.510520 22911 raft_consensus.cc:729] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3790ec5d5aa34650b987539520e6d120, State: Initialized, Role: FOLLOWER
I20260927 15:25:04.511463 22911 consensus_queue.cc:260] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.512939 22889 heartbeater.cc:503] Master 127.2.246.126:37375 was elected leader, sending a full tablet report...
I20260927 15:25:04.513808 22911 ts_tablet_manager.cc:1428] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260927 15:25:04.579905 22890 tablet.cc:2369] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:04.593993 22916 raft_consensus.cc:482] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:25:04.594354 22916 raft_consensus.cc:504] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.596267 22916 leader_election.cc:288] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2f69e2cef74a8cb97af3e5eefe9626 (127.2.246.65:45215), 47f2b60c190449119e92c1428e039c49 (127.2.246.66:43701)
I20260927 15:25:04.603582 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.597070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:45215 (local address 127.2.246.67:57735)
0927 15:25:04.597379 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.597405 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.597575 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.598243 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.598251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.598291 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:04.598587 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.598604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.599887 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.599893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.601435 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.601455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.602934 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.602942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.603126 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.603153 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.603240 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.603346 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 15:25:04.603822 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.597183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:57735 (local address 127.2.246.65:45215)
0927 15:25:04.597455 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.597460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.597492 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:25:04.597803 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.597809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.597974 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.598153 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:04.598823 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.599712 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.601697 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.602741 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.602918 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.603036 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.603296 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.603463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:25:04.605254 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d46c69d585c4d5eb872d4c79fe09292" candidate_uuid: "3790ec5d5aa34650b987539520e6d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" is_pre_election: true
I20260927 15:25:04.605767 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.597640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:43701 (local address 127.2.246.67:43621)
0927 15:25:04.598860 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:04.598876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:04.599012 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:04.599392 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:04.599396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:04.599421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:04.599737 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.599745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.601348 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.601354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.602869 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:04.602886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.604782 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.604793 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:04.605004 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.605042 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:25:04.605164 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 15:25:04.605331 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":583,"threads_started":1}
I20260927 15:25:04.605976 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.597122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:43621 (local address 127.2.246.66:43701)
0927 15:25:04.597404 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.597411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.598966 (+ 1555us) server_negotiation.cc:407] Connection header received
0927 15:25:04.599153 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.599159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.599269 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.599375 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:04.600051 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.601135 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.603111 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.604463 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.604680 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.604774 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.605259 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.605673 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 15:25:04.606068 22576 raft_consensus.cc:2454] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3790ec5d5aa34650b987539520e6d120 in term 0.
I20260927 15:25:04.606985 22777 leader_election.cc:302] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2f69e2cef74a8cb97af3e5eefe9626, 3790ec5d5aa34650b987539520e6d120; no voters:
I20260927 15:25:04.607048 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d46c69d585c4d5eb872d4c79fe09292" candidate_uuid: "3790ec5d5aa34650b987539520e6d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f2b60c190449119e92c1428e039c49" is_pre_election: true
I20260927 15:25:04.607568 22916 raft_consensus.cc:2781] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:25:04.607609 22710 raft_consensus.cc:2454] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3790ec5d5aa34650b987539520e6d120 in term 0.
I20260927 15:25:04.607703 22916 raft_consensus.cc:482] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:25:04.607864 22916 raft_consensus.cc:3037] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:04.610812 22916 raft_consensus.cc:504] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.611786 22916 leader_election.cc:288] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2f69e2cef74a8cb97af3e5eefe9626 (127.2.246.65:45215), 47f2b60c190449119e92c1428e039c49 (127.2.246.66:43701)
I20260927 15:25:04.612454 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d46c69d585c4d5eb872d4c79fe09292" candidate_uuid: "3790ec5d5aa34650b987539520e6d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626"
I20260927 15:25:04.612654 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d46c69d585c4d5eb872d4c79fe09292" candidate_uuid: "3790ec5d5aa34650b987539520e6d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f2b60c190449119e92c1428e039c49"
I20260927 15:25:04.612720 22576 raft_consensus.cc:3037] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:04.612946 22710 raft_consensus.cc:3037] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:04.615470 22576 raft_consensus.cc:2454] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3790ec5d5aa34650b987539520e6d120 in term 1.
I20260927 15:25:04.615842 22710 raft_consensus.cc:2454] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3790ec5d5aa34650b987539520e6d120 in term 1.
I20260927 15:25:04.616142 22777 leader_election.cc:302] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2f69e2cef74a8cb97af3e5eefe9626, 3790ec5d5aa34650b987539520e6d120; no voters:
I20260927 15:25:04.616571 22916 raft_consensus.cc:2781] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:04.617233 22916 raft_consensus.cc:686] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 1 LEADER]: Becoming Leader. State: Replica: 3790ec5d5aa34650b987539520e6d120, State: Running, Role: LEADER
I20260927 15:25:04.617986 22916 consensus_queue.cc:237] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } }
I20260927 15:25:04.624246 22431 catalog_manager.cc:5495] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3790ec5d5aa34650b987539520e6d120 (127.2.246.67). New cstate: current_term: 1 leader_uuid: "3790ec5d5aa34650b987539520e6d120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3790ec5d5aa34650b987539520e6d120" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 39187 } health_report { overall_health: HEALTHY } } }
W20260927 15:25:04.642426 22622 tablet.cc:2369] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:04.659356 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.652680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.2.246.67:39187)
0927 15:25:04.652871 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.652876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.653161 (+ 285us) server_negotiation.cc:407] Connection header received
0927 15:25:04.653370 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.653374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.653498 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.653654 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:04.654203 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.655147 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.656717 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.657194 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.657355 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.658084 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:04.658686 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:04.658843 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.658995 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.659077 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:25:04.670693 22576 raft_consensus.cc:1261] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 1 FOLLOWER]: Refusing update from remote peer 3790ec5d5aa34650b987539520e6d120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:04.671142 22710 raft_consensus.cc:1261] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 1 FOLLOWER]: Refusing update from remote peer 3790ec5d5aa34650b987539520e6d120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:04.671957 22918 consensus_queue.cc:1035] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2f69e2cef74a8cb97af3e5eefe9626" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:04.672466 22916 consensus_queue.cc:1035] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f2b60c190449119e92c1428e039c49" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:04.685909 22922 mvcc.cc:205] Tried to move back new op lower bound from 7333980998317031424 to 7333980998120951808. Current Snapshot: MvccSnapshot[applied={T|T < 7333980998317031424}]
I20260927 15:25:04.705756 22431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:25:04.706274 22431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:25:04.711153 22431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 677ba9abe99e4a4a9b5f4187cb15a0d3: Sending DeleteTablet for 3 replicas of tablet 4d46c69d585c4d5eb872d4c79fe09292
I20260927 15:25:04.712828 22690 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:04 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:50124
I20260927 15:25:04.713052 22824 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:04 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:48862
I20260927 15:25:04.712972 22556 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d46c69d585c4d5eb872d4c79fe09292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:04 UTC) from {username='kudu', principal='kudu/127.2.246.126@KRBTEST.COM'} at 127.0.0.1:36750
I20260927 15:25:04.716118 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22493
I20260927 15:25:04.724939 22931 tablet_replica.cc:330] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: stopping tablet replica
I20260927 15:25:04.726007 22931 raft_consensus.cc:2229] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:25:04.727227 22931 raft_consensus.cc:2258] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:04.727893 22932 tablet_replica.cc:330] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: stopping tablet replica
I20260927 15:25:04.728588 22932 raft_consensus.cc:2229] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:25:04.732334 22931 ts_tablet_manager.cc:1905] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:25:04.733978 22932 raft_consensus.cc:2258] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:04.734935 22930 tablet_replica.cc:330] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: stopping tablet replica
I20260927 15:25:04.735644 22930 raft_consensus.cc:2229] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:25:04.736505 22930 raft_consensus.cc:2258] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:04.737623 22932 ts_tablet_manager.cc:1905] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:25:04 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.65@KRBTEST.COM
I20260927 15:25:04.739616 22930 ts_tablet_manager.cc:1905] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:25:04.749143 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.718442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.2.246.65:45215)
0927 15:25:04.718746 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.718757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.722208 (+ 3451us) server_negotiation.cc:407] Connection header received
0927 15:25:04.722574 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.722585 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.722760 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.722903 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:04.728938 (+ 6035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.730468 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.732882 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.734029 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.734210 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.740877 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:04.740899 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:04.740919 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:04.740984 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:04.745162 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.746342 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.746352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.746355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.746530 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.747014 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.747019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.747023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.747505 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:04.747816 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.748448 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.748710 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:25:04.749096 22930 ts_tablet_manager.cc:1918] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:25:04.749593 22930 log.cc:1196] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/4d46c69d585c4d5eb872d4c79fe09292
I20260927 15:25:04.750313 22930 ts_tablet_manager.cc:1939] T 4d46c69d585c4d5eb872d4c79fe09292 P 47f2b60c190449119e92c1428e039c49: Deleting consensus metadata
I20260927 15:25:04.751861 22616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:04.755920 22932 ts_tablet_manager.cc:1918] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:25:04.756207 22932 log.cc:1196] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/4d46c69d585c4d5eb872d4c79fe09292
I20260927 15:25:04.756790 22932 ts_tablet_manager.cc:1939] T 4d46c69d585c4d5eb872d4c79fe09292 P 3790ec5d5aa34650b987539520e6d120: Deleting consensus metadata
I20260927 15:25:04.759074 22418 catalog_manager.cc:4841] TS 3790ec5d5aa34650b987539520e6d120 (127.2.246.67:39187): tablet 4d46c69d585c4d5eb872d4c79fe09292 (table test-table [id=e1dbb75640e14e73b6d418bdf9463df2]) successfully deleted
I20260927 15:25:04.760461 22418 catalog_manager.cc:4841] TS 47f2b60c190449119e92c1428e039c49 (127.2.246.66:43701): tablet 4d46c69d585c4d5eb872d4c79fe09292 (table test-table [id=e1dbb75640e14e73b6d418bdf9463df2]) successfully deleted
I20260927 15:25:04.867573 22931 ts_tablet_manager.cc:1918] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:25:04.867834 22931 log.cc:1196] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522351366233-3033-0/minicluster-data/ts-0/wal/wals/4d46c69d585c4d5eb872d4c79fe09292
I20260927 15:25:04.868392 22931 ts_tablet_manager.cc:1939] T 4d46c69d585c4d5eb872d4c79fe09292 P 0a2f69e2cef74a8cb97af3e5eefe9626: Deleting consensus metadata
I20260927 15:25:04.870985 22416 catalog_manager.cc:4841] TS 0a2f69e2cef74a8cb97af3e5eefe9626 (127.2.246.65:45215): tablet 4d46c69d585c4d5eb872d4c79fe09292 (table test-table [id=e1dbb75640e14e73b6d418bdf9463df2]) successfully deleted
I20260927 15:25:04.872381 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22493
I20260927 15:25:04.896692 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22627
Sep 27 15:25:04 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.66@KRBTEST.COM
I20260927 15:25:04.917514 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:04.897812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.2.246.66:43701)
0927 15:25:04.898121 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:04.898129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:04.898188 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:25:04.898267 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:04.898275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:04.898378 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:04.898494 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:04.900198 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.901372 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.903087 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:04.903683 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:04.903856 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:04.909209 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:04.909229 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:04.909250 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:04.909317 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:04.913297 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.914744 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.914756 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.914759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.914974 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:04.915435 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:04.915439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:04.915442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:04.915947 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:04.916314 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:04.916661 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:25:04.917071 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 15:25:04.918509 22750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:05.028563 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22627
I20260927 15:25:05.050609 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22761
Sep 27 15:25:05 dist-test-slave-14jq krb5kdc[22387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.67@KRBTEST.COM
I20260927 15:25:05.074910 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:05.051741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.2.246.67:39187)
0927 15:25:05.051964 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:05.051969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:05.052100 (+ 131us) server_negotiation.cc:407] Connection header received
0927 15:25:05.052276 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:05.052281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:05.052417 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:05.052547 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:05.054624 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:05.055899 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:05.058031 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:05.058646 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:05.058811 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:05.065311 (+ 6500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:05.065334 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:05.065357 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:05.065448 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:05.070623 (+ 5175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:05.071980 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:05.071990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:05.071993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:05.072226 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:05.072744 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:05.072749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:05.072751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:05.073298 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:05.073758 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:05.074093 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:25:05.074405 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:25:05.075788 22884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:05.234127 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22761
I20260927 15:25:05.256345 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22400
I20260927 15:25:05.257917 22462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:05.377987 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22400
2026-09-27T15:25:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5001 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:05Z Disabled control of system clock
I20260927 15:25:05.476190 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42549
--webserver_interface=127.2.246.126
--webserver_port=0
--builtin_ntp_servers=127.2.246.84:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:25:05.832114 22941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:05.832434 22941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:05.832517 22941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:05.839974 22941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:05.840399 22941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:05.840471 22941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:05.840526 22941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:05.840584 22941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:05.850720 22941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:05.852224 22941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:05.854149 22941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:05.868073 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:05.868320 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:05.868551 22946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:05.869622 22941 server_base.cc:1034] running on GCE node
I20260927 15:25:05.870643 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:05.872370 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:05.873627 22941 hybrid_clock.cc:648] HybridClock initialized: now 1790522705873544 us; error 86 us; skew 500 ppm
I20260927 15:25:05.874151 22941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:25:05.877074 22941 webserver.cc:460] Webserver started at http://127.2.246.126:36179/ using document root <none> and password file <none>
I20260927 15:25:05.877952 22941 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:05.878057 22941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:05.878398 22941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:05.881534 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "cac60592860449b0b48f442324faa942"
format_stamp: "Formatted at 2026-09-27 15:25:05 on dist-test-slave-14jq"
server_key: "d419434574a3a0cf0963582ced285462"
server_key_iv: "59afdedac0ed81d895651d52e585dc5e"
server_key_version: "encryptionkey@0"
I20260927 15:25:05.882575 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "cac60592860449b0b48f442324faa942"
format_stamp: "Formatted at 2026-09-27 15:25:05 on dist-test-slave-14jq"
server_key: "d419434574a3a0cf0963582ced285462"
server_key_iv: "59afdedac0ed81d895651d52e585dc5e"
server_key_version: "encryptionkey@0"
I20260927 15:25:05.888625 22941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:25:05.893148 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:05.895129 22941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:25:05.895373 22941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "cac60592860449b0b48f442324faa942"
format_stamp: "Formatted at 2026-09-27 15:25:05 on dist-test-slave-14jq"
server_key: "d419434574a3a0cf0963582ced285462"
server_key_iv: "59afdedac0ed81d895651d52e585dc5e"
server_key_version: "encryptionkey@0"
I20260927 15:25:05.895629 22941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:05.918617 22941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:05.919638 22941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:05.919993 22941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:05.942631 22941 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:42549
I20260927 15:25:05.942655 23006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:42549 every 8 connection(s)
I20260927 15:25:05.944132 22941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:05.950075 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:05.954432 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 22941
I20260927 15:25:05.954710 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:05.955339 3033 external_mini_cluster.cc:1428] Setting key fe33696f5e898ae523497206c7027e48
I20260927 15:25:05.969281 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: Bootstrap starting.
I20260927 15:25:05.974542 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:05.958329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.2.246.126:42549)
0927 15:25:05.959966 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:05.959980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:05.960037 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:25:05.962371 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:05.962402 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:05.963019 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:05.963459 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:05.964441 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:05.966153 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:05.968399 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:05.969322 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:05.969621 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:05.971290 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:05.971299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:05.971330 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:05.972054 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:05.972156 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:05.972185 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:05.972326 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:05.972726 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:25:05.973275 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":569,"threads_started":1}
I20260927 15:25:05.975008 23007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:05.976667 23007 log.cc:824] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:05.981271 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: No bootstrap required, opened a new log
I20260927 15:25:05.990916 23007 raft_consensus.cc:348] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } }
I20260927 15:25:05.991523 23007 raft_consensus.cc:374] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:05.991652 23007 raft_consensus.cc:729] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac60592860449b0b48f442324faa942, State: Initialized, Role: FOLLOWER
I20260927 15:25:05.992424 23007 consensus_queue.cc:260] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } }
I20260927 15:25:05.992756 23007 raft_consensus.cc:388] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:05.992925 23007 raft_consensus.cc:482] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:05.993130 23007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:05.996322 23007 raft_consensus.cc:504] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } }
I20260927 15:25:05.997155 23007 leader_election.cc:302] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cac60592860449b0b48f442324faa942; no voters:
I20260927 15:25:05.998253 23007 leader_election.cc:288] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:05.998358 23012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:05.999611 23012 raft_consensus.cc:686] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [term 1 LEADER]: Becoming Leader. State: Replica: cac60592860449b0b48f442324faa942, State: Running, Role: LEADER
I20260927 15:25:06.000574 23012 consensus_queue.cc:237] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } }
I20260927 15:25:06.001821 23007 sys_catalog.cc:564] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:06.003826 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac60592860449b0b48f442324faa942. Latest consensus state: current_term: 1 leader_uuid: "cac60592860449b0b48f442324faa942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } } }
I20260927 15:25:06.004341 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:06.005178 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cac60592860449b0b48f442324faa942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac60592860449b0b48f442324faa942" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42549 } } }
I20260927 15:25:06.005604 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:06.009088 23017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:06.018078 23017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:06.030575 23017 catalog_manager.cc:1347] Generated new cluster ID: 208c0be50d7d4adcb17a47e8f56c0e0d
I20260927 15:25:06.030751 23017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:06.048123 23017 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:06.050232 23017 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:06.064992 23017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: Generated new TSK 0
I20260927 15:25:06.066331 23017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:06.077013 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.65:0
--local_ip_for_outbound_sockets=127.2.246.65
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--builtin_ntp_servers=127.2.246.84:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:25:06.549664 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:06.550203 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:06.550396 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:06.559988 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:06.560261 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:06.560475 23031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.65
I20260927 15:25:06.573452 23031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.65
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.65
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:06.575363 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:06.577773 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:06.595486 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:06.595867 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:06.596627 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:06.597453 23031 server_base.cc:1034] running on GCE node
I20260927 15:25:06.598435 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:06.599929 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:06.601310 23031 hybrid_clock.cc:648] HybridClock initialized: now 1790522706601209 us; error 83 us; skew 500 ppm
I20260927 15:25:06.601807 23031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:25:06.605404 23031 webserver.cc:460] Webserver started at http://127.2.246.65:41037/ using document root <none> and password file <none>
I20260927 15:25:06.606364 23031 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:06.606477 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:06.606834 23031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:06.610615 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data/instance:
uuid: "89342ccac0aa4a24bce1d348be08fbc9"
format_stamp: "Formatted at 2026-09-27 15:25:06 on dist-test-slave-14jq"
server_key: "a9f687ccd6a390f57c821f3400934ff7"
server_key_iv: "c8bfb4e8f69cc59b358c2757e7c02400"
server_key_version: "encryptionkey@0"
I20260927 15:25:06.611589 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance:
uuid: "89342ccac0aa4a24bce1d348be08fbc9"
format_stamp: "Formatted at 2026-09-27 15:25:06 on dist-test-slave-14jq"
server_key: "a9f687ccd6a390f57c821f3400934ff7"
server_key_iv: "c8bfb4e8f69cc59b358c2757e7c02400"
server_key_version: "encryptionkey@0"
I20260927 15:25:06.617669 23031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:25:06.623674 23045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:06.626188 23031 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:25:06.626431 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal
uuid: "89342ccac0aa4a24bce1d348be08fbc9"
format_stamp: "Formatted at 2026-09-27 15:25:06 on dist-test-slave-14jq"
server_key: "a9f687ccd6a390f57c821f3400934ff7"
server_key_iv: "c8bfb4e8f69cc59b358c2757e7c02400"
server_key_version: "encryptionkey@0"
I20260927 15:25:06.626801 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:06.660889 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:06.661819 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:06.662215 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:06.663492 23031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:06.665335 23031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:06.665477 23031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:06.665611 23031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:06.665695 23031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:06.707605 23031 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.65:40937
I20260927 15:25:06.707772 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.65:40937 every 8 connection(s)
I20260927 15:25:06.709022 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/data/info.pb
I20260927 15:25:06.716336 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23031
I20260927 15:25:06.716581 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-0/wal/instance
I20260927 15:25:06.717113 3033 external_mini_cluster.cc:1428] Setting key 83dcade6fc89badf56a8351e2ab965dd
I20260927 15:25:06.721028 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.66:0
--local_ip_for_outbound_sockets=127.2.246.66
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--builtin_ntp_servers=127.2.246.84:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:25:06.727124 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:06.712812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.65:33609 (local address 127.2.246.126:42549)
0927 15:25:06.713357 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:06.713362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:06.714952 (+ 1590us) server_negotiation.cc:407] Connection header received
0927 15:25:06.717072 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:06.717078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:06.717179 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:06.717327 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:06.719250 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:06.720159 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:06.723276 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:06.723738 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:06.723898 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:06.726203 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:06.726208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:06.726212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:06.726237 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:06.726296 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:06.726309 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:06.726393 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:06.726723 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:25:06.726830 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260927 15:25:06.729171 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:06.713427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42549 (local address 127.2.246.65:33609)
0927 15:25:06.714758 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:06.714805 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:06.716777 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:06.717818 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:06.717871 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:06.718042 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:06.719003 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:06.719034 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:06.720354 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:06.720368 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:06.723029 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:06.723067 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:06.724113 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:06.724124 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:06.724556 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:06.725606 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:25:06.725612 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:06.725691 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:25:06.725871 (+ 180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:06.726431 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:06.726445 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:06.726458 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 15:25:06.726688 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 15:25:06.727308 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":491,"threads_started":1}
I20260927 15:25:06.731465 23158 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42549
I20260927 15:25:06.731962 23158 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:06.733023 23158 heartbeater.cc:511] Master 127.2.246.126:42549 requested a full tablet report, sending...
I20260927 15:25:06.735788 22972 ts_manager.cc:194] Registered new tserver with Master: 89342ccac0aa4a24bce1d348be08fbc9 (127.2.246.65:40937)
I20260927 15:25:06.738097 22972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.65:33609
W20260927 15:25:07.095132 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:07.095455 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:07.095556 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:07.104933 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:07.105170 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:07.105443 23162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.66
I20260927 15:25:07.115700 23162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.66
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.66
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:07.117054 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:07.119099 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:07.135154 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:07.135136 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:07.135658 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:07.136449 23162 server_base.cc:1034] running on GCE node
I20260927 15:25:07.137219 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:07.138482 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:07.139845 23162 hybrid_clock.cc:648] HybridClock initialized: now 1790522707139780 us; error 48 us; skew 500 ppm
I20260927 15:25:07.140256 23162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:25:07.143327 23162 webserver.cc:460] Webserver started at http://127.2.246.66:41475/ using document root <none> and password file <none>
I20260927 15:25:07.144177 23162 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:07.144325 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:07.144685 23162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:07.148058 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data/instance:
uuid: "672fa4f828d54cb3a65c399941319136"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "4f9a06029776d9c03282fbc16e1918f0"
server_key_iv: "945007056bce9f7d20a1a77c04970eb6"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.148983 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance:
uuid: "672fa4f828d54cb3a65c399941319136"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "4f9a06029776d9c03282fbc16e1918f0"
server_key_iv: "945007056bce9f7d20a1a77c04970eb6"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.155306 23162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:25:07.159955 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.162071 23162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:25:07.162317 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal
uuid: "672fa4f828d54cb3a65c399941319136"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "4f9a06029776d9c03282fbc16e1918f0"
server_key_iv: "945007056bce9f7d20a1a77c04970eb6"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.162598 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:07.183871 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:07.184823 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:07.185168 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:07.186360 23162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:07.188409 23162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:07.188577 23162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.188740 23162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:07.188845 23162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.232640 23162 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.66:33035
I20260927 15:25:07.232709 23288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.66:33035 every 8 connection(s)
I20260927 15:25:07.234217 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/data/info.pb
I20260927 15:25:07.240676 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23162
I20260927 15:25:07.240939 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal/instance
I20260927 15:25:07.241453 3033 external_mini_cluster.cc:1428] Setting key 65b02c28bd5cf3ea18a8d1eb443332da
I20260927 15:25:07.245574 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.67:0
--local_ip_for_outbound_sockets=127.2.246.67
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--builtin_ntp_servers=127.2.246.84:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:25:07.250375 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.238094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:59947 (local address 127.2.246.126:42549)
0927 15:25:07.238700 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.238706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.240623 (+ 1917us) server_negotiation.cc:407] Connection header received
0927 15:25:07.243029 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.243034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.243114 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.243229 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:07.244608 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.245355 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.247211 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.247658 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.247815 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.249402 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:07.249407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:07.249410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:07.249433 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:07.249474 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:07.249486 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:07.249584 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.249875 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.249997 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":307,"threads_started":1}
I20260927 15:25:07.251471 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.238909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42549 (local address 127.2.246.66:59947)
0927 15:25:07.240381 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:07.240440 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:07.242594 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:07.243566 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:07.243580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:07.243735 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:07.244415 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.244434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.245546 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.245551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.247023 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.247039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.247941 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.247951 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.248212 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.249013 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:25:07.249016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:07.249061 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:25:07.249161 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:07.249599 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:07.249615 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:07.249627 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:25:07.249831 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 15:25:07.250316 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":397,"threads_started":1}
I20260927 15:25:07.253713 23289 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42549
I20260927 15:25:07.254125 23289 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:07.255031 23289 heartbeater.cc:511] Master 127.2.246.126:42549 requested a full tablet report, sending...
I20260927 15:25:07.256748 22972 ts_manager.cc:194] Registered new tserver with Master: 672fa4f828d54cb3a65c399941319136 (127.2.246.66:33035)
I20260927 15:25:07.257604 22972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.66:59947
W20260927 15:25:07.605373 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:07.605726 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:07.605940 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:07.613564 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:07.613722 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:07.613976 23293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.67
I20260927 15:25:07.624962 23293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.67
--webserver_port=0
--tserver_master_addrs=127.2.246.126:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.67
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:07.626546 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:07.628545 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:07.647056 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:07.647118 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:07.648015 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:07.652078 23293 server_base.cc:1034] running on GCE node
I20260927 15:25:07.652874 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:07.654160 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:07.655438 23293 hybrid_clock.cc:648] HybridClock initialized: now 1790522707655370 us; error 64 us; skew 500 ppm
I20260927 15:25:07.655925 23293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:25:07.659004 23293 webserver.cc:460] Webserver started at http://127.2.246.67:36429/ using document root <none> and password file <none>
I20260927 15:25:07.659806 23293 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:07.659919 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:07.660264 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:07.663623 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data/instance:
uuid: "66ad0ce149e0452b86f96d61a74e6c05"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "020e2c6a5aba283e4354b09b9c053d69"
server_key_iv: "d21972ae2e0f4b2594ed966c4a795710"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.664533 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance:
uuid: "66ad0ce149e0452b86f96d61a74e6c05"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "020e2c6a5aba283e4354b09b9c053d69"
server_key_iv: "d21972ae2e0f4b2594ed966c4a795710"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.670790 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 15:25:07.675191 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.676946 23293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:25:07.677193 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal
uuid: "66ad0ce149e0452b86f96d61a74e6c05"
format_stamp: "Formatted at 2026-09-27 15:25:07 on dist-test-slave-14jq"
server_key: "020e2c6a5aba283e4354b09b9c053d69"
server_key_iv: "d21972ae2e0f4b2594ed966c4a795710"
server_key_version: "encryptionkey@0"
I20260927 15:25:07.677497 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:07.701812 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:07.702878 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:07.703199 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:07.704409 23293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:25:07.706487 23293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:25:07.706620 23293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.706754 23293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:25:07.706846 23293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:07.742820 23158 heartbeater.cc:503] Master 127.2.246.126:42549 was elected leader, sending a full tablet report...
I20260927 15:25:07.749500 23293 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.67:42681
I20260927 15:25:07.749660 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.67:42681 every 8 connection(s)
I20260927 15:25:07.751077 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/data/info.pb
I20260927 15:25:07.753134 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23293
I20260927 15:25:07.753432 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal/instance
I20260927 15:25:07.754138 3033 external_mini_cluster.cc:1428] Setting key 2824064070900214697e9ab1b62f1743
I20260927 15:25:07.770979 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.755439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.67:53775 (local address 127.2.246.126:42549)
0927 15:25:07.756152 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.756159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.758234 (+ 2075us) server_negotiation.cc:407] Connection header received
0927 15:25:07.760820 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.760826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.760911 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.761047 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:07.762855 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.763716 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.766790 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.767340 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.767519 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.769706 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:07.769713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:07.769717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:07.769745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:07.769799 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:07.769813 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:07.769971 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.770337 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.770489 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260927 15:25:07.772521 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.756076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42549 (local address 127.2.246.67:53775)
0927 15:25:07.757936 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:07.758011 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:07.760370 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:07.761421 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:07.761433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:07.761591 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:07.762559 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.762592 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.763962 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.763967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.766486 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.766529 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.767853 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.767860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.768218 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.769156 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:25:07.769159 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:07.769218 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:25:07.769346 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:07.769977 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:07.769992 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:07.770003 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:25:07.770285 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 15:25:07.770878 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":472,"threads_started":1}
I20260927 15:25:07.774576 23420 heartbeater.cc:347] Connected to a master server at 127.2.246.126:42549
I20260927 15:25:07.775128 23420 heartbeater.cc:464] Registering TS with master...
I20260927 15:25:07.776233 23420 heartbeater.cc:511] Master 127.2.246.126:42549 requested a full tablet report, sending...
I20260927 15:25:07.778376 22972 ts_manager.cc:194] Registered new tserver with Master: 66ad0ce149e0452b86f96d61a74e6c05 (127.2.246.67:42681)
I20260927 15:25:07.779271 22972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.67:53775
I20260927 15:25:07.785392 3033 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:25:07.797535 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.790418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.2.246.126:42549)
0927 15:25:07.790730 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.790738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.790882 (+ 144us) server_negotiation.cc:407] Connection header received
0927 15:25:07.791198 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.791204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.791283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.791408 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:07.791983 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.792789 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.794355 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.794819 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.794972 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.796275 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:07.796281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:07.796284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:07.796311 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:07.796367 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:07.796382 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:07.796497 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.796828 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.796992 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 15:25:07.803385 22972 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:25:07.811676 22972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:25:07.848968 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.828010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:42681 (local address 127.0.0.1:39318)
0927 15:25:07.829017 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:07.829071 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:07.829336 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:07.840356 (+ 11020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:07.840366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:07.840401 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:07.840972 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.841006 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.843510 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.843515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.845539 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.845565 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.847680 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.847686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.847972 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.848039 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 15:25:07.848224 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 15:25:07.848501 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":36,"thread_start_us":432,"threads_started":1}
I20260927 15:25:07.848956 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.828348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.2.246.67:42681)
0927 15:25:07.839543 (+ 11195us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.839549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.839583 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:25:07.839718 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.839728 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.840057 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.840315 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:07.841318 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.843254 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.845865 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.847420 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.847606 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.847969 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.848298 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.848511 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11026,"thread_start_us":10620,"threads_started":1}
I20260927 15:25:07.849603 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.828040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.66:33035 (local address 127.0.0.1:36822)
0927 15:25:07.839538 (+ 11498us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:07.839571 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:07.839820 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:07.840785 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:07.840793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:07.840836 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:07.841384 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.841399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.843437 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.843444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.846234 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.846252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.848703 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.848709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.848897 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.848935 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:25:07.849034 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:25:07.849177 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11281,"mutex_wait_us":103,"thread_start_us":11775,"threads_started":1}
I20260927 15:25:07.851235 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.841447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54528)
0927 15:25:07.842087 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:07.842106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:07.842230 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:07.843040 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:07.843046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:07.843071 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:07.843499 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.843512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.846373 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.846380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.848450 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:07.848465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.850450 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.850457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:07.850624 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.850656 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:25:07.850794 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 15:25:07.850921 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
I20260927 15:25:07.851642 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.840722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.2.246.65:40937)
0927 15:25:07.841529 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.841539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.842376 (+ 837us) server_negotiation.cc:407] Connection header received
0927 15:25:07.842497 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.842505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.842820 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.843022 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:07.843848 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.846183 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.848687 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.850241 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.850422 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.850706 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.851041 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.851238 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":359,"threads_started":1}
I20260927 15:25:07.853240 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:07.829671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.2.246.66:33035)
0927 15:25:07.838622 (+ 8951us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:07.838637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:07.839746 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 15:25:07.840106 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:07.840122 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:07.840421 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:07.840648 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:07.841643 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.843259 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.846507 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:07.848135 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:07.848284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:07.848567 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:07.849456 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 15:25:07.849684 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8636,"thread_start_us":8480,"threads_started":1}
I20260927 15:25:07.854970 23355 tablet_service.cc:1467] Processing CreateTablet for tablet 542c4582777d4e9bbab81192c1f71396 (DEFAULT_TABLE table=test-table [id=754251dedb5e4669861574b7ea7bb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:07.855701 23093 tablet_service.cc:1467] Processing CreateTablet for tablet 542c4582777d4e9bbab81192c1f71396 (DEFAULT_TABLE table=test-table [id=754251dedb5e4669861574b7ea7bb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:07.856964 23355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c4582777d4e9bbab81192c1f71396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:07.857097 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c4582777d4e9bbab81192c1f71396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:07.856853 23224 tablet_service.cc:1467] Processing CreateTablet for tablet 542c4582777d4e9bbab81192c1f71396 (DEFAULT_TABLE table=test-table [id=754251dedb5e4669861574b7ea7bb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:25:07.858383 23224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c4582777d4e9bbab81192c1f71396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:07.872462 23439 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Bootstrap starting.
I20260927 15:25:07.875932 23439 tablet_bootstrap.cc:654] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:07.876631 23441 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Bootstrap starting.
I20260927 15:25:07.877503 23439 log.cc:824] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:07.880908 23440 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Bootstrap starting.
I20260927 15:25:07.883522 23441 tablet_bootstrap.cc:654] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:07.884893 23439 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: No bootstrap required, opened a new log
I20260927 15:25:07.885057 23440 tablet_bootstrap.cc:654] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:07.885044 23441 log.cc:824] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:07.885298 23439 ts_tablet_manager.cc:1397] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260927 15:25:07.886241 23440 log.cc:824] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:07.889439 23441 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: No bootstrap required, opened a new log
I20260927 15:25:07.889810 23441 ts_tablet_manager.cc:1397] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 15:25:07.893976 23440 tablet_bootstrap.cc:492] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: No bootstrap required, opened a new log
I20260927 15:25:07.894369 23440 ts_tablet_manager.cc:1397] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260927 15:25:07.897400 23439 raft_consensus.cc:348] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.898201 23439 raft_consensus.cc:374] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:07.898030 23441 raft_consensus.cc:348] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.898387 23439 raft_consensus.cc:729] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672fa4f828d54cb3a65c399941319136, State: Initialized, Role: FOLLOWER
I20260927 15:25:07.898550 23441 raft_consensus.cc:374] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:07.898679 23441 raft_consensus.cc:729] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89342ccac0aa4a24bce1d348be08fbc9, State: Initialized, Role: FOLLOWER
I20260927 15:25:07.899513 23441 consensus_queue.cc:260] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.899451 23439 consensus_queue.cc:260] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.901559 23441 ts_tablet_manager.cc:1428] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260927 15:25:07.902596 23289 heartbeater.cc:503] Master 127.2.246.126:42549 was elected leader, sending a full tablet report...
I20260927 15:25:07.903816 23439 ts_tablet_manager.cc:1428] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Time spent starting tablet: real 0.018s user 0.013s sys 0.008s
I20260927 15:25:07.907495 23440 raft_consensus.cc:348] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.908183 23440 raft_consensus.cc:374] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:07.908313 23440 raft_consensus.cc:729] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ad0ce149e0452b86f96d61a74e6c05, State: Initialized, Role: FOLLOWER
I20260927 15:25:07.909111 23440 consensus_queue.cc:260] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:07.910563 23420 heartbeater.cc:503] Master 127.2.246.126:42549 was elected leader, sending a full tablet report...
I20260927 15:25:07.911402 23440 ts_tablet_manager.cc:1428] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
W20260927 15:25:07.962199 23159 tablet.cc:2369] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:25:07.987126 23290 tablet.cc:2369] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:25:08.003746 23421 tablet.cc:2369] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:08.022547 23446 raft_consensus.cc:482] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:25:08.022962 23446 raft_consensus.cc:504] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:08.024928 23446 leader_election.cc:288] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ad0ce149e0452b86f96d61a74e6c05 (127.2.246.67:42681), 89342ccac0aa4a24bce1d348be08fbc9 (127.2.246.65:40937)
I20260927 15:25:08.035138 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.026096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:49915 (local address 127.2.246.65:40937)
0927 15:25:08.026303 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.026310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.026658 (+ 348us) server_negotiation.cc:407] Connection header received
0927 15:25:08.026921 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.026926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.026994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.027073 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:08.027856 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.028845 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.032294 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.033705 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.033917 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.033995 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.034679 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.034836 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:25:08.035809 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.025543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.2.246.66:49915)
0927 15:25:08.026524 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.026541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:08.026757 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:08.027142 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:08.027148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:08.027176 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:08.027591 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:08.027599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.029017 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.029022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:08.031904 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:08.031927 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.034079 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.034088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:08.034398 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.034489 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 15:25:08.034618 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:25:08.034843 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":250,"thread_start_us":582,"threads_started":1}
I20260927 15:25:08.035959 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.025547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.246.66:33133 (local address 127.2.246.67:42681)
0927 15:25:08.025894 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.025904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.026179 (+ 275us) server_negotiation.cc:407] Connection header received
0927 15:25:08.026533 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.026538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.026631 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.026743 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:25:08.027532 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.028702 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.032287 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.033676 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.033917 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.033995 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.035434 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.035600 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 15:25:08.036065 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.025144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.67:42681 (local address 127.2.246.66:33133)
0927 15:25:08.026032 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.026054 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:08.026345 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:08.026794 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:08.026803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:08.026833 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:25:08.027332 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:08.027350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.028878 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.028883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:08.031911 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:08.031927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.034079 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.034088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:08.034414 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.034485 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:25:08.034633 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:25:08.034843 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":286,"thread_start_us":428,"threads_started":1}
I20260927 15:25:08.037407 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c4582777d4e9bbab81192c1f71396" candidate_uuid: "672fa4f828d54cb3a65c399941319136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89342ccac0aa4a24bce1d348be08fbc9" is_pre_election: true
I20260927 15:25:08.038056 23113 raft_consensus.cc:2454] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672fa4f828d54cb3a65c399941319136 in term 0.
I20260927 15:25:08.038344 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c4582777d4e9bbab81192c1f71396" candidate_uuid: "672fa4f828d54cb3a65c399941319136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ad0ce149e0452b86f96d61a74e6c05" is_pre_election: true
I20260927 15:25:08.038908 23375 raft_consensus.cc:2454] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672fa4f828d54cb3a65c399941319136 in term 0.
I20260927 15:25:08.038954 23179 leader_election.cc:302] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672fa4f828d54cb3a65c399941319136, 89342ccac0aa4a24bce1d348be08fbc9; no voters:
I20260927 15:25:08.039463 23446 raft_consensus.cc:2781] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:25:08.039628 23446 raft_consensus.cc:482] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:25:08.039754 23446 raft_consensus.cc:3037] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:08.042110 23446 raft_consensus.cc:504] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:08.042786 23446 leader_election.cc:288] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [CANDIDATE]: Term 1 election: Requested vote from peers 66ad0ce149e0452b86f96d61a74e6c05 (127.2.246.67:42681), 89342ccac0aa4a24bce1d348be08fbc9 (127.2.246.65:40937)
I20260927 15:25:08.043622 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c4582777d4e9bbab81192c1f71396" candidate_uuid: "672fa4f828d54cb3a65c399941319136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ad0ce149e0452b86f96d61a74e6c05"
I20260927 15:25:08.043877 23375 raft_consensus.cc:3037] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:08.043845 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c4582777d4e9bbab81192c1f71396" candidate_uuid: "672fa4f828d54cb3a65c399941319136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89342ccac0aa4a24bce1d348be08fbc9"
I20260927 15:25:08.044126 23113 raft_consensus.cc:3037] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:08.046473 23375 raft_consensus.cc:2454] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672fa4f828d54cb3a65c399941319136 in term 1.
I20260927 15:25:08.047111 23113 raft_consensus.cc:2454] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672fa4f828d54cb3a65c399941319136 in term 1.
I20260927 15:25:08.047137 23180 leader_election.cc:302] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ad0ce149e0452b86f96d61a74e6c05, 672fa4f828d54cb3a65c399941319136; no voters:
I20260927 15:25:08.047561 23446 raft_consensus.cc:2781] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:08.048192 23446 raft_consensus.cc:686] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 1 LEADER]: Becoming Leader. State: Replica: 672fa4f828d54cb3a65c399941319136, State: Running, Role: LEADER
I20260927 15:25:08.048918 23446 consensus_queue.cc:237] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } }
I20260927 15:25:08.054536 22972 catalog_manager.cc:5495] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672fa4f828d54cb3a65c399941319136 (127.2.246.66). New cstate: current_term: 1 leader_uuid: "672fa4f828d54cb3a65c399941319136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672fa4f828d54cb3a65c399941319136" member_type: VOTER last_known_addr { host: "127.2.246.66" port: 33035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 } health_report { overall_health: UNKNOWN } } }
I20260927 15:25:08.098248 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.090941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.2.246.66:33035)
0927 15:25:08.091265 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.091271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.091325 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:25:08.091452 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.091457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.091583 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.091768 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:08.092271 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.093115 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.094908 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.095618 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.095784 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.096416 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:08.097086 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:08.097217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.097659 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.097811 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 15:25:08.108677 23113 raft_consensus.cc:1261] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 1 FOLLOWER]: Refusing update from remote peer 672fa4f828d54cb3a65c399941319136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:08.108971 23375 raft_consensus.cc:1261] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 1 FOLLOWER]: Refusing update from remote peer 672fa4f828d54cb3a65c399941319136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:08.110069 23450 consensus_queue.cc:1035] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89342ccac0aa4a24bce1d348be08fbc9" member_type: VOTER last_known_addr { host: "127.2.246.65" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:08.110831 23446 consensus_queue.cc:1035] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ad0ce149e0452b86f96d61a74e6c05" member_type: VOTER last_known_addr { host: "127.2.246.67" port: 42681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:08.125203 23454 mvcc.cc:205] Tried to move back new op lower bound from 7333981012399517696 to 7333981012173697024. Current Snapshot: MvccSnapshot[applied={T|T < 7333981012399517696}]
I20260927 15:25:08.134204 23456 mvcc.cc:205] Tried to move back new op lower bound from 7333981012399517696 to 7333981012173697024. Current Snapshot: MvccSnapshot[applied={T|T < 7333981012399517696}]
I20260927 15:25:08.142414 22971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:25:08.142709 22971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:25:08.147840 22971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cac60592860449b0b48f442324faa942: Sending DeleteTablet for 3 replicas of tablet 542c4582777d4e9bbab81192c1f71396
I20260927 15:25:08.149506 23355 tablet_service.cc:1514] Processing DeleteTablet for tablet 542c4582777d4e9bbab81192c1f71396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:08 UTC) from {username='slave'} at 127.0.0.1:39318
I20260927 15:25:08.149603 23093 tablet_service.cc:1514] Processing DeleteTablet for tablet 542c4582777d4e9bbab81192c1f71396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:08 UTC) from {username='slave'} at 127.0.0.1:54528
I20260927 15:25:08.150254 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23031
I20260927 15:25:08.152418 23224 tablet_service.cc:1514] Processing DeleteTablet for tablet 542c4582777d4e9bbab81192c1f71396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:25:08 UTC) from {username='slave'} at 127.0.0.1:36822
I20260927 15:25:08.159148 23463 tablet_replica.cc:330] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: stopping tablet replica
I20260927 15:25:08.161005 23463 raft_consensus.cc:2229] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:25:08.162686 23463 raft_consensus.cc:2258] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:08.164462 23466 tablet_replica.cc:330] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: stopping tablet replica
I20260927 15:25:08.165153 23466 raft_consensus.cc:2229] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:25:08.166133 23466 raft_consensus.cc:2258] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:08.168248 23464 tablet_replica.cc:330] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: stopping tablet replica
I20260927 15:25:08.168743 23464 raft_consensus.cc:2229] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:25:08.169308 23464 raft_consensus.cc:2258] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:25:08.170397 23463 ts_tablet_manager.cc:1905] T 542c4582777d4e9bbab81192c1f71396 P 89342ccac0aa4a24bce1d348be08fbc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:25:08.171705 23466 ts_tablet_manager.cc:1905] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:25:08.175560 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.151505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.2.246.65:40937)
0927 15:25:08.151682 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.151688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.166979 (+ 15291us) server_negotiation.cc:407] Connection header received
0927 15:25:08.167218 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.167226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.167349 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.167458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:08.168160 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.169278 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.171217 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.171838 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.171941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.173557 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:08.173571 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:08.173592 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:08.174518 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:08.174592 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:08.174619 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:08.174726 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.175015 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.175153 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:25:08.177099 23153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:08.179915 23464 ts_tablet_manager.cc:1905] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:25:08.183918 23466 ts_tablet_manager.cc:1918] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:25:08.184521 23466 log.cc:1196] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-1/wal/wals/542c4582777d4e9bbab81192c1f71396
I20260927 15:25:08.185411 23466 ts_tablet_manager.cc:1939] T 542c4582777d4e9bbab81192c1f71396 P 672fa4f828d54cb3a65c399941319136: Deleting consensus metadata
I20260927 15:25:08.188287 22958 catalog_manager.cc:4841] TS 672fa4f828d54cb3a65c399941319136 (127.2.246.66:33035): tablet 542c4582777d4e9bbab81192c1f71396 (table test-table [id=754251dedb5e4669861574b7ea7bb52c]) successfully deleted
I20260927 15:25:08.189950 23464 ts_tablet_manager.cc:1918] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:25:08.190196 23464 log.cc:1196] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Deleting WAL directory at /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522351366233-3033-0/minicluster-data/ts-2/wal/wals/542c4582777d4e9bbab81192c1f71396
I20260927 15:25:08.190815 23464 ts_tablet_manager.cc:1939] T 542c4582777d4e9bbab81192c1f71396 P 66ad0ce149e0452b86f96d61a74e6c05: Deleting consensus metadata
I20260927 15:25:08.193651 22959 catalog_manager.cc:4841] TS 66ad0ce149e0452b86f96d61a74e6c05 (127.2.246.67:42681): tablet 542c4582777d4e9bbab81192c1f71396 (table test-table [id=754251dedb5e4669861574b7ea7bb52c]) successfully deleted
I20260927 15:25:08.281874 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23031
I20260927 15:25:08.311106 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23162
W20260927 15:25:08.311093 22958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.246.65:40937 (error 108)
I20260927 15:25:08.313122 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.312624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54552)
0927 15:25:08.312827 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.312947 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 15:25:08.313429 22958 catalog_manager.cc:4584] TS 89342ccac0aa4a24bce1d348be08fbc9 (127.2.246.65:40937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 542c4582777d4e9bbab81192c1f71396: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
I20260927 15:25:08.320847 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.312185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.2.246.66:33035)
0927 15:25:08.312441 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.312449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.312477 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:25:08.312683 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.312691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.312829 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.312999 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:08.313488 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.314952 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.316371 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.317152 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.317340 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.318692 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:08.318706 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:08.318731 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:08.319767 (+ 1036us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:08.319847 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:08.319882 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:08.320025 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.320223 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.320381 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:25:08.321808 23284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:08.367583 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.366827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54566)
0927 15:25:08.367176 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.367325 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 15:25:08.369069 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.368597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54582)
0927 15:25:08.368790 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.368889 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 15:25:08.430471 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.429739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54592)
0927 15:25:08.430073 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.430215 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 15:25:08.431795 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.431336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54604)
0927 15:25:08.431558 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.431651 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:25:08.442386 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23162
I20260927 15:25:08.462773 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23293
I20260927 15:25:08.470269 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.463784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.2.246.67:42681)
0927 15:25:08.464015 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:08.464022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:08.464109 (+ 87us) server_negotiation.cc:407] Connection header received
0927 15:25:08.464356 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:08.464361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:08.464446 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:08.464560 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:08.464965 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.465675 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.466966 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:08.467369 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:08.467494 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:08.468789 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:08.468806 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:08.468821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:08.469469 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:08.469516 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:08.469536 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:08.469658 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:08.469800 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:25:08.469930 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 15:25:08.471359 23415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:08.512486 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.511748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54618)
0927 15:25:08.512070 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.512233 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 15:25:08.514102 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:08.513537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.65:40937 (local address 127.0.0.1:54624)
0927 15:25:08.513777 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:08.513948 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.65:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 15:25:08.566123 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23293
I20260927 15:25:08.583879 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 22941
I20260927 15:25:08.585460 23002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:08.701870 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 22941
2026-09-27T15:25:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3321 ms)
[----------] 6 tests from EncryptionPolicyTest (25224 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:08 dist-test-slave-14jq krb5kdc[23474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:08 dist-test-slave-14jq krb5kdc[23474](info): set up 2 sockets
Sep 27 15:25:08 dist-test-slave-14jq krb5kdc[23474](info): commencing operation
krb5kdc: starting...
W20260927 15:25:09.874660 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:10 dist-test-slave-14jq krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:10.182615 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:10.534842 23487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:10.535169 23487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:10.535305 23487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:10.543112 23487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:10.543264 23487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:10.543314 23487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:10.543418 23487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:10.543524 23487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:10.554180 23487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:10.555635 23487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:10.557049 23487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:10.570303 23495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:10.570750 23492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:10.570575 23493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:10.572342 23487 server_base.cc:1034] running on GCE node
I20260927 15:25:10.573693 23487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:10.575834 23487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:10.577149 23487 hybrid_clock.cc:648] HybridClock initialized: now 1790522710577088 us; error 52 us; skew 500 ppm
I20260927 15:25:10.577665 23487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:10 dist-test-slave-14jq krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:10.582270 23487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:10.584805 23487 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260927 15:25:10.585825 23487 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:10.586095 23487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:10.586508 23487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:10.590788 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "8427ffae558d43529c274d694d40c967"
format_stamp: "Formatted at 2026-09-27 15:25:10 on dist-test-slave-14jq"
server_key: "451f6e54e7d652d017561327a7bde821"
server_key_iv: "42f814428128b81fc2240c9beea447fe"
server_key_version: "encryptionkey@0"
I20260927 15:25:10.591917 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "8427ffae558d43529c274d694d40c967"
format_stamp: "Formatted at 2026-09-27 15:25:10 on dist-test-slave-14jq"
server_key: "451f6e54e7d652d017561327a7bde821"
server_key_iv: "42f814428128b81fc2240c9beea447fe"
server_key_version: "encryptionkey@0"
I20260927 15:25:10.600415 23487 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260927 15:25:10.606072 23502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:10.607982 23487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:25:10.608265 23487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "8427ffae558d43529c274d694d40c967"
format_stamp: "Formatted at 2026-09-27 15:25:10 on dist-test-slave-14jq"
server_key: "451f6e54e7d652d017561327a7bde821"
server_key_iv: "42f814428128b81fc2240c9beea447fe"
server_key_version: "encryptionkey@0"
I20260927 15:25:10.608548 23487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:10.633299 23487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:10.640972 23487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:10.641431 23487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:10.668040 23487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46847
I20260927 15:25:10.668056 23553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46847 every 8 connection(s)
I20260927 15:25:10.670181 23487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:10.674656 23554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:10.675884 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23487
I20260927 15:25:10.676177 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:10.676903 3033 external_mini_cluster.cc:1428] Setting key 6f35447ecdfc78fa3d7c390d8d97c20b
I20260927 15:25:10.688665 23554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967: Bootstrap starting.
I20260927 15:25:10.692965 23554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:10.694514 23554 log.cc:824] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:25:10 dist-test-slave-14jq krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:25:10.698844 23554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967: No bootstrap required, opened a new log
I20260927 15:25:10.707935 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:10.679412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48525 (local address 127.0.0.1:46847)
0927 15:25:10.680531 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:10.680546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:10.680620 (+ 74us) server_negotiation.cc:407] Connection header received
0927 15:25:10.683182 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:10.683220 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:10.683842 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:10.684329 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:10.686294 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:10.687899 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:10.689885 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:10.690834 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:10.691085 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:10.697475 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:10.697487 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:10.697512 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:10.697568 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:10.701798 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:10.703000 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:10.703009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:10.703019 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:10.703264 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:10.704091 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:10.704109 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:10.704112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:10.704955 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:10.705377 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:10.706050 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 15:25:10.706486 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":535,"threads_started":1}
I20260927 15:25:10.707693 23554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } }
I20260927 15:25:10.708204 23554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:10.708335 23554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8427ffae558d43529c274d694d40c967, State: Initialized, Role: FOLLOWER
I20260927 15:25:10.709110 23554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } }
I20260927 15:25:10.709384 23554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:10.709545 23554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:10.709738 23554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:10.712934 23554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } }
I20260927 15:25:10.713581 23554 leader_election.cc:302] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8427ffae558d43529c274d694d40c967; no voters:
I20260927 15:25:10.714022 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:10.714481 23554 leader_election.cc:288] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:10.714711 23559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:10.716930 23554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:10.721930 23559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [term 1 LEADER]: Becoming Leader. State: Replica: 8427ffae558d43529c274d694d40c967, State: Running, Role: LEADER
I20260927 15:25:10.723018 23559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } }
I20260927 15:25:10.727007 23560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8427ffae558d43529c274d694d40c967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } } }
I20260927 15:25:10.727752 23560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:10.727308 23561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8427ffae558d43529c274d694d40c967. Latest consensus state: current_term: 1 leader_uuid: "8427ffae558d43529c274d694d40c967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8427ffae558d43529c274d694d40c967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46847 } } }
I20260927 15:25:10.728360 23561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:10.729353 23574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:10.741596 23574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:10.750768 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:10.726343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36437 (local address 127.0.0.1:46847)
0927 15:25:10.727127 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:10.727135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:10.728190 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 15:25:10.728487 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:10.728494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:10.728622 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:10.728722 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:10.731330 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:10.733244 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:10.735683 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:10.736488 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:10.736606 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:10.741281 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:10.741295 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:10.741304 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:10.741370 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:10.746055 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:10.747299 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:10.747304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:10.747307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:10.747483 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:10.748071 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:10.748076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:10.748079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:10.748568 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:10.748842 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:10.750165 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0927 15:25:10.750373 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606}
I20260927 15:25:10.759550 23574 catalog_manager.cc:1347] Generated new cluster ID: 50dfe8751434498bb870a044f35ea803
I20260927 15:25:10.759724 23574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:10.783577 23574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:10.785516 23574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:10.793994 23574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8427ffae558d43529c274d694d40c967: Generated new TSK 0
I20260927 15:25:10.795096 23574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:10.814740 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23487
I20260927 15:25:10.817621 23549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:10.927284 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23487
W20260927 15:25:10.946125 23472 connection.cc:535] client connection to 127.0.0.1:46847 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46847): Connection reset by peer (error 104)
2026-09-27T15:25:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:11 dist-test-slave-14jq krb5kdc[23586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:11 dist-test-slave-14jq krb5kdc[23586](info): set up 2 sockets
Sep 27 15:25:11 dist-test-slave-14jq krb5kdc[23586](info): commencing operation
krb5kdc: starting...
W20260927 15:25:12.134582 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:12 dist-test-slave-14jq krb5kdc[23586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:12.442102 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:12.870586 23599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:12.871111 23599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:12.871363 23599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:12.880978 23599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:12.881184 23599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:12.881326 23599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:12.881448 23599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:12.881522 23599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:12.894706 23599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:12.896772 23599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:12.898589 23599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:12.915148 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:12.915220 23604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:12.915247 23607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:12.916119 23599 server_base.cc:1034] running on GCE node
I20260927 15:25:12.917344 23599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:12.919459 23599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:12.920745 23599 hybrid_clock.cc:648] HybridClock initialized: now 1790522712920667 us; error 63 us; skew 500 ppm
I20260927 15:25:12.921298 23599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:12 dist-test-slave-14jq krb5kdc[23586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:12.925506 23599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:12.927688 23599 webserver.cc:460] Webserver started at http://127.0.0.1:39365/ using document root <none> and password file <none>
I20260927 15:25:12.928678 23599 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:12.928854 23599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:12.929239 23599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:12.933460 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "4f362302ce514cbfbf3bbe9c70f32750"
format_stamp: "Formatted at 2026-09-27 15:25:12 on dist-test-slave-14jq"
server_key: "23d8c6167081bf3f5e9e5bc49ef137b2"
server_key_iv: "2399dbc08f3a36ea0e96a165c8879259"
server_key_version: "encryptionkey@0"
I20260927 15:25:12.934711 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "4f362302ce514cbfbf3bbe9c70f32750"
format_stamp: "Formatted at 2026-09-27 15:25:12 on dist-test-slave-14jq"
server_key: "23d8c6167081bf3f5e9e5bc49ef137b2"
server_key_iv: "2399dbc08f3a36ea0e96a165c8879259"
server_key_version: "encryptionkey@0"
I20260927 15:25:12.942407 23599 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 15:25:12.947562 23614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:12.949589 23599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 15:25:12.949941 23599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "4f362302ce514cbfbf3bbe9c70f32750"
format_stamp: "Formatted at 2026-09-27 15:25:12 on dist-test-slave-14jq"
server_key: "23d8c6167081bf3f5e9e5bc49ef137b2"
server_key_iv: "2399dbc08f3a36ea0e96a165c8879259"
server_key_version: "encryptionkey@0"
I20260927 15:25:12.950243 23599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:12.971125 23599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:12.977260 23599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:12.977641 23599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:13.003470 23599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46575
I20260927 15:25:13.003515 23665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46575 every 8 connection(s)
I20260927 15:25:13.005527 23599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:13.010124 23666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:13.015137 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23599
I20260927 15:25:13.015465 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:13.015987 3033 external_mini_cluster.cc:1428] Setting key 09f2ec3c5aab951574b471eeb4db1d98
I20260927 15:25:13.021919 23666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750: Bootstrap starting.
I20260927 15:25:13.025606 23666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:13.026950 23666 log.cc:824] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:13.031140 23666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750: No bootstrap required, opened a new log
I20260927 15:25:13.034256 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:13.018607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60923 (local address 127.0.0.1:46575)
0927 15:25:13.019743 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:13.019761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:13.019817 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:25:13.021968 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:13.021998 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:13.022570 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:13.022981 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:13.023743 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:13.025224 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:13.027264 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:13.028713 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:13.028952 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:13.030796 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:13.030806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:13.030827 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:13.031643 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:13.031769 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:13.031803 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:13.032072 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:13.032361 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:25:13.032702 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":558,"threads_started":1}
I20260927 15:25:13.039773 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:13.040679 23666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 15:25:13.041114 23666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:13.041222 23666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f362302ce514cbfbf3bbe9c70f32750, State: Initialized, Role: FOLLOWER
I20260927 15:25:13.041903 23666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 15:25:13.042160 23666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:13.042304 23666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:13.042486 23666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:13.045075 23666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 15:25:13.045802 23666 leader_election.cc:302] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f362302ce514cbfbf3bbe9c70f32750; no voters:
I20260927 15:25:13.046968 23666 leader_election.cc:288] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:13.047129 23676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:13.049183 23676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [term 1 LEADER]: Becoming Leader. State: Replica: 4f362302ce514cbfbf3bbe9c70f32750, State: Running, Role: LEADER
I20260927 15:25:13.050171 23676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 15:25:13.050839 23666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:13.056046 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f362302ce514cbfbf3bbe9c70f32750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } } }
I20260927 15:25:13.056526 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:13.057626 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:13.046035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52063 (local address 127.0.0.1:46575)
0927 15:25:13.046309 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:13.046318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:13.046865 (+ 547us) server_negotiation.cc:407] Connection header received
0927 15:25:13.047244 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:13.047263 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:13.047384 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:13.048405 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:13.048635 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:13.050048 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:13.052283 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:13.053356 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:13.053536 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:13.055800 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:13.055811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:13.055820 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:13.055859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:13.055974 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:13.056007 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:13.056175 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:13.057002 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 15:25:13.057093 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 15:25:13.061180 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f362302ce514cbfbf3bbe9c70f32750. Latest consensus state: current_term: 1 leader_uuid: "4f362302ce514cbfbf3bbe9c70f32750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f362302ce514cbfbf3bbe9c70f32750" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } } }
I20260927 15:25:13.061705 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:13.064000 23683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:13.070173 23683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:13.081815 23683 catalog_manager.cc:1347] Generated new cluster ID: 4e745d0caa2f42498ea903e5ff517cef
I20260927 15:25:13.082050 23683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:13.092118 23683 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:13.094177 23683 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:13.103230 23683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f362302ce514cbfbf3bbe9c70f32750: Generated new TSK 0
I20260927 15:25:13.104333 23683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:13.114953 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23599
W20260927 15:25:13.116464 23661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60923
W20260927 15:25:13.117504 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:13.117751 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23599
W20260927 15:25:13.139240 23584 connection.cc:535] client connection to 127.0.0.1:46575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46575): Connection reset by peer (error 104)
2026-09-27T15:25:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:13 dist-test-slave-14jq krb5kdc[23697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:13 dist-test-slave-14jq krb5kdc[23697](info): set up 2 sockets
Sep 27 15:25:13 dist-test-slave-14jq krb5kdc[23697](info): commencing operation
krb5kdc: starting...
W20260927 15:25:14.267052 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:14 dist-test-slave-14jq krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:14.580189 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:14.941383 23710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:14.941694 23710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:14.941788 23710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:14.949294 23710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:14.949431 23710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:14.949479 23710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:14.949604 23710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:14.949674 23710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:14.959970 23710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:14.961367 23710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:14.962749 23710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:14.976086 23718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:14.976219 23715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:14.976257 23716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:14.977320 23710 server_base.cc:1034] running on GCE node
I20260927 15:25:14.978305 23710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:14.979928 23710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:14.981210 23710 hybrid_clock.cc:648] HybridClock initialized: now 1790522714981150 us; error 47 us; skew 500 ppm
I20260927 15:25:14.981694 23710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:14 dist-test-slave-14jq krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:14.989573 23710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:14.991493 23710 webserver.cc:460] Webserver started at http://127.0.0.1:34973/ using document root <none> and password file <none>
I20260927 15:25:14.992282 23710 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:14.992409 23710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:14.992704 23710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:14.996155 23710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "632f9c0aa04e42d2862dca015da59aed"
format_stamp: "Formatted at 2026-09-27 15:25:14 on dist-test-slave-14jq"
server_key: "353f2b7643942108227aee9544fc4264"
server_key_iv: "b049bcbb70f64695e057cf57dc8db346"
server_key_version: "encryptionkey@0"
I20260927 15:25:14.996978 23710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "632f9c0aa04e42d2862dca015da59aed"
format_stamp: "Formatted at 2026-09-27 15:25:14 on dist-test-slave-14jq"
server_key: "353f2b7643942108227aee9544fc4264"
server_key_iv: "b049bcbb70f64695e057cf57dc8db346"
server_key_version: "encryptionkey@0"
I20260927 15:25:15.003574 23710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:25:15.008071 23725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:15.010164 23710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:25:15.010411 23710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "632f9c0aa04e42d2862dca015da59aed"
format_stamp: "Formatted at 2026-09-27 15:25:14 on dist-test-slave-14jq"
server_key: "353f2b7643942108227aee9544fc4264"
server_key_iv: "b049bcbb70f64695e057cf57dc8db346"
server_key_version: "encryptionkey@0"
I20260927 15:25:15.010661 23710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:15.017045 23710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:15.021709 23710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:15.022146 23710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:15.043881 23710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36529
I20260927 15:25:15.044020 23776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36529 every 8 connection(s)
I20260927 15:25:15.045498 23710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:15.050448 23777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:15.052361 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23710
I20260927 15:25:15.052592 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:15.053254 3033 external_mini_cluster.cc:1428] Setting key 1f15015c69be0b220850c4bf6ed6684e
I20260927 15:25:15.063494 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:15.055678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50829 (local address 127.0.0.1:36529)
0927 15:25:15.056822 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:15.056839 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:15.056897 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:25:15.059165 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:15.059197 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:15.059780 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:15.060277 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:15.061564 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:15.061572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:15.061586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:15.061602 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:15.061671 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:15.061698 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:15.062010 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:15.062184 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:25:15.062402 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":447,"threads_started":1}
I20260927 15:25:15.068100 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:15.068120 23777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed: Bootstrap starting.
I20260927 15:25:15.072860 23777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:15.075634 23777 log.cc:824] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:15.076602 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:15.072872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52223 (local address 127.0.0.1:36529)
0927 15:25:15.073081 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:15.073087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:15.073452 (+ 365us) server_negotiation.cc:407] Connection header received
0927 15:25:15.073709 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:15.073713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:15.073773 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:15.073960 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:15.075611 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:15.075615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:15.075619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:15.075634 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:15.075677 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:15.075693 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:15.075932 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:15.076118 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:25:15.076190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:25:15.080006 23777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed: No bootstrap required, opened a new log
I20260927 15:25:15.088266 23777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } }
I20260927 15:25:15.088676 23777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:15.088807 23777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632f9c0aa04e42d2862dca015da59aed, State: Initialized, Role: FOLLOWER
I20260927 15:25:15.089463 23777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } }
I20260927 15:25:15.089681 23777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:15.089898 23777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:15.090047 23777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:15.092398 23777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } }
I20260927 15:25:15.092948 23777 leader_election.cc:302] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632f9c0aa04e42d2862dca015da59aed; no voters:
I20260927 15:25:15.093739 23777 leader_election.cc:288] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:15.093986 23787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:15.094906 23787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [term 1 LEADER]: Becoming Leader. State: Replica: 632f9c0aa04e42d2862dca015da59aed, State: Running, Role: LEADER
I20260927 15:25:15.095592 23787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } }
I20260927 15:25:15.096266 23777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:15.098978 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "632f9c0aa04e42d2862dca015da59aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } } }
I20260927 15:25:15.099494 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:15.099898 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632f9c0aa04e42d2862dca015da59aed. Latest consensus state: current_term: 1 leader_uuid: "632f9c0aa04e42d2862dca015da59aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632f9c0aa04e42d2862dca015da59aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36529 } } }
I20260927 15:25:15.100343 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:15.107548 23793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:15.115523 23793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:15.128558 23793 catalog_manager.cc:1347] Generated new cluster ID: ed918dd767ac4b5d82eaa24e1df8618c
I20260927 15:25:15.128746 23793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:15.139411 23793 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:15.139881 23793 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:15.149492 23793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632f9c0aa04e42d2862dca015da59aed: Generated new TSK 0
I20260927 15:25:15.150467 23793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:15.164655 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23710
W20260927 15:25:15.165912 23772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50829
W20260927 15:25:15.166688 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23710: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:15.166837 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23710
2026-09-27T15:25:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:15 dist-test-slave-14jq krb5kdc[23808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:15 dist-test-slave-14jq krb5kdc[23808](info): set up 2 sockets
Sep 27 15:25:15 dist-test-slave-14jq krb5kdc[23808](info): commencing operation
krb5kdc: starting...
W20260927 15:25:16.310629 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:16 dist-test-slave-14jq krb5kdc[23808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:16.618371 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:16.966130 23821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:16.966446 23821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:16.966578 23821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:16.974100 23821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:16.974226 23821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:16.974272 23821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:16.974359 23821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:16.974431 23821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:16.984536 23821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:16.985936 23821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:16.987272 23821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:17.001482 23827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:17.001645 23826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:17.001952 23829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:17.003197 23821 server_base.cc:1034] running on GCE node
I20260927 15:25:17.004443 23821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:17.006651 23821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:17.007964 23821 hybrid_clock.cc:648] HybridClock initialized: now 1790522717007899 us; error 64 us; skew 500 ppm
I20260927 15:25:17.008487 23821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:17 dist-test-slave-14jq krb5kdc[23808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:17.013031 23821 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:17.015237 23821 webserver.cc:460] Webserver started at http://127.0.0.1:39505/ using document root <none> and password file <none>
I20260927 15:25:17.016150 23821 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:17.016275 23821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:17.016583 23821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:17.020571 23821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "c6d289876c194d38963803411e5f9b15"
format_stamp: "Formatted at 2026-09-27 15:25:17 on dist-test-slave-14jq"
server_key: "5feb7b53066818bc45c773230a1132e6"
server_key_iv: "917a8419072645e225a49e814e154399"
server_key_version: "encryptionkey@0"
I20260927 15:25:17.021569 23821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "c6d289876c194d38963803411e5f9b15"
format_stamp: "Formatted at 2026-09-27 15:25:17 on dist-test-slave-14jq"
server_key: "5feb7b53066818bc45c773230a1132e6"
server_key_iv: "917a8419072645e225a49e814e154399"
server_key_version: "encryptionkey@0"
I20260927 15:25:17.029139 23821 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 15:25:17.034484 23836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:17.036517 23821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 15:25:17.036806 23821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "c6d289876c194d38963803411e5f9b15"
format_stamp: "Formatted at 2026-09-27 15:25:17 on dist-test-slave-14jq"
server_key: "5feb7b53066818bc45c773230a1132e6"
server_key_iv: "917a8419072645e225a49e814e154399"
server_key_version: "encryptionkey@0"
I20260927 15:25:17.037225 23821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:17.063426 23821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:17.069442 23821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:17.069921 23821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:17.093914 23821 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34685
I20260927 15:25:17.093955 23887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34685 every 8 connection(s)
I20260927 15:25:17.095820 23821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:17.100214 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:17.100997 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23821
I20260927 15:25:17.101212 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:17.101890 3033 external_mini_cluster.cc:1428] Setting key 75c151792c4232966fed5909203b18cc
I20260927 15:25:17.115911 23888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15: Bootstrap starting.
I20260927 15:25:17.119490 23888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15: Neither blocks nor log segments found. Creating new log.
Sep 27 15:25:17 dist-test-slave-14jq krb5kdc[23808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:25:17.120848 23888 log.cc:824] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:17.125321 23888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15: No bootstrap required, opened a new log
I20260927 15:25:17.132465 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:17.104394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36013 (local address 127.0.0.1:34685)
0927 15:25:17.105469 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:17.105486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:17.105540 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:25:17.107707 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:17.107747 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:17.108395 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:17.108900 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:17.110754 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:17.112828 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:17.114970 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:17.116088 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:17.117070 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:17.122241 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:17.122254 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:17.122277 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:17.122340 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:17.126499 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:17.127873 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:17.127883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:17.127994 (+ 111us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:17.128282 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:17.128932 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:17.128937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:17.128943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:17.129700 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:17.130127 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:17.130612 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 15:25:17.131135 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":487,"threads_started":1}
I20260927 15:25:17.136874 23888 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 15:25:17.137506 23888 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:17.137669 23888 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d289876c194d38963803411e5f9b15, State: Initialized, Role: FOLLOWER
I20260927 15:25:17.137993 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:17.138504 23888 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 15:25:17.138818 23888 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:17.138990 23888 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:17.139369 23888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:17.142745 23888 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 15:25:17.143774 23888 leader_election.cc:302] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d289876c194d38963803411e5f9b15; no voters:
I20260927 15:25:17.144704 23888 leader_election.cc:288] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:17.144863 23898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:17.146870 23898 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [term 1 LEADER]: Becoming Leader. State: Replica: c6d289876c194d38963803411e5f9b15, State: Running, Role: LEADER
I20260927 15:25:17.147778 23898 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 15:25:17.148072 23888 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:17.152524 23899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d289876c194d38963803411e5f9b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260927 15:25:17.153191 23899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:17.154151 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d289876c194d38963803411e5f9b15. Latest consensus state: current_term: 1 leader_uuid: "c6d289876c194d38963803411e5f9b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d289876c194d38963803411e5f9b15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260927 15:25:17.155118 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:17.166594 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:17.143207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53447 (local address 127.0.0.1:34685)
0927 15:25:17.143508 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:17.143519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:17.144070 (+ 551us) server_negotiation.cc:407] Connection header received
0927 15:25:17.144298 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:17.144303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:17.144445 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:17.144552 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:17.146611 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:17.147627 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:17.149043 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:17.149561 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:17.149698 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:17.153340 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:17.153346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:17.153351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:17.153408 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:17.159283 (+ 5875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:17.160572 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:17.160581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:17.160588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:17.160880 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:17.161672 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:17.161681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:17.161687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:17.162409 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:17.162990 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:17.165464 (+ 2474us) server_negotiation.cc:299] Negotiation successful
0927 15:25:17.165947 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:25:17.169629 23905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:17.176367 23905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:17.188265 23905 catalog_manager.cc:1347] Generated new cluster ID: 1f1f6066a3f945c0b2b4aeb4b57b7790
I20260927 15:25:17.188413 23905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:17.220469 23905 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:17.222268 23905 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:17.242327 23905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d289876c194d38963803411e5f9b15: Generated new TSK 0
I20260927 15:25:17.243685 23905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:17.261780 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23821
I20260927 15:25:17.263368 23883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:17.382526 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23821
2026-09-27T15:25:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:17 dist-test-slave-14jq krb5kdc[23921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:17 dist-test-slave-14jq krb5kdc[23921](info): set up 2 sockets
Sep 27 15:25:17 dist-test-slave-14jq krb5kdc[23921](info): commencing operation
krb5kdc: starting...
W20260927 15:25:18.558967 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:18 dist-test-slave-14jq krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:18.879643 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:19.248157 23934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:19.248556 23934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:19.248726 23934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:19.256651 23934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:19.256858 23934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:19.256950 23934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:19.257045 23934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:19.257139 23934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:19.268014 23934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:19.269464 23934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:19.270871 23934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:19.285056 23939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:19.285152 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:19.285172 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:19.286309 23934 server_base.cc:1034] running on GCE node
I20260927 15:25:19.287225 23934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:19.288954 23934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:19.290428 23934 hybrid_clock.cc:648] HybridClock initialized: now 1790522719290368 us; error 56 us; skew 500 ppm
I20260927 15:25:19.290912 23934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:19 dist-test-slave-14jq krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:19.294675 23934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:19.296756 23934 webserver.cc:460] Webserver started at http://127.0.0.1:37261/ using document root <none> and password file <none>
I20260927 15:25:19.297580 23934 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:19.297757 23934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:19.298188 23934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:19.302178 23934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "63e9cceb5d054d7eb06c2d629cb199c0"
format_stamp: "Formatted at 2026-09-27 15:25:19 on dist-test-slave-14jq"
server_key: "594b36bfae8ff22688c02f3d9d8435bf"
server_key_iv: "adcd227371f2fe1ab6bffbc70249adb2"
server_key_version: "encryptionkey@0"
I20260927 15:25:19.303295 23934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "63e9cceb5d054d7eb06c2d629cb199c0"
format_stamp: "Formatted at 2026-09-27 15:25:19 on dist-test-slave-14jq"
server_key: "594b36bfae8ff22688c02f3d9d8435bf"
server_key_iv: "adcd227371f2fe1ab6bffbc70249adb2"
server_key_version: "encryptionkey@0"
I20260927 15:25:19.310398 23934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:25:19.315045 23949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:19.317059 23934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:25:19.317337 23934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "63e9cceb5d054d7eb06c2d629cb199c0"
format_stamp: "Formatted at 2026-09-27 15:25:19 on dist-test-slave-14jq"
server_key: "594b36bfae8ff22688c02f3d9d8435bf"
server_key_iv: "adcd227371f2fe1ab6bffbc70249adb2"
server_key_version: "encryptionkey@0"
I20260927 15:25:19.317574 23934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:19.356549 23934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:19.361722 23934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:19.362100 23934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:19.387495 23934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45925
I20260927 15:25:19.387447 24000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45925 every 8 connection(s)
I20260927 15:25:19.389478 23934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:19.395334 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 23934
I20260927 15:25:19.395596 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:19.396267 3033 external_mini_cluster.cc:1428] Setting key 73611c9584a5d80ca2ea0517b7ae1f95
I20260927 15:25:19.397023 24001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:19.414495 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:19.398914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54827 (local address 127.0.0.1:45925)
0927 15:25:19.400194 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:19.400212 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:19.400269 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:25:19.403797 (+ 3528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:19.403833 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:19.404401 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:19.404986 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:19.405637 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:19.407292 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:19.408847 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:19.409623 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:19.409895 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:19.411227 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:19.411236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:19.411256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:19.411925 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:19.412028 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:19.412060 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:19.412220 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:19.412647 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 15:25:19.413078 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":541,"threads_started":1}
I20260927 15:25:19.416604 24001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0: Bootstrap starting.
I20260927 15:25:19.420514 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:19.422079 24001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:19.423765 24001 log.cc:824] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:19.429136 24001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0: No bootstrap required, opened a new log
I20260927 15:25:19.435437 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:19.427049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55781 (local address 127.0.0.1:45925)
0927 15:25:19.427275 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:19.427281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:19.427687 (+ 406us) server_negotiation.cc:407] Connection header received
0927 15:25:19.428040 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:19.428045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:19.428128 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:19.428262 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:19.429019 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:19.430101 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:19.432308 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:19.432911 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:19.433063 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:19.434657 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:19.434663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:19.434667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:19.434689 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:19.434752 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:19.434772 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:19.434864 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:19.435050 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:25:19.435149 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:25:19.442004 24001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260927 15:25:19.442620 24001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:19.442720 24001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e9cceb5d054d7eb06c2d629cb199c0, State: Initialized, Role: FOLLOWER
I20260927 15:25:19.443542 24001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260927 15:25:19.443941 24001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:19.444129 24001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:19.444376 24001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:19.447404 24001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260927 15:25:19.448176 24001 leader_election.cc:302] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e9cceb5d054d7eb06c2d629cb199c0; no voters:
I20260927 15:25:19.448979 24001 leader_election.cc:288] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:19.449378 24011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:19.450366 24011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [term 1 LEADER]: Becoming Leader. State: Replica: 63e9cceb5d054d7eb06c2d629cb199c0, State: Running, Role: LEADER
I20260927 15:25:19.451134 24011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260927 15:25:19.452215 24001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:19.454722 24013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e9cceb5d054d7eb06c2d629cb199c0. Latest consensus state: current_term: 1 leader_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260927 15:25:19.455574 24013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:19.456845 24012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9cceb5d054d7eb06c2d629cb199c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260927 15:25:19.458051 24012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:19.468662 24020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:19.474428 24020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:19.486227 24020 catalog_manager.cc:1347] Generated new cluster ID: 1c1999f297984d70bb4ef71df553dafd
I20260927 15:25:19.486406 24020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:19.497891 24020 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:19.499351 24020 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:19.506431 24020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e9cceb5d054d7eb06c2d629cb199c0: Generated new TSK 0
I20260927 15:25:19.507583 24020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:19.529152 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23934
W20260927 15:25:19.530598 23996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54827
W20260927 15:25:19.531436 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 23934: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:19.531921 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 23934
2026-09-27T15:25:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:19 dist-test-slave-14jq krb5kdc[24032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:19 dist-test-slave-14jq krb5kdc[24032](info): set up 2 sockets
Sep 27 15:25:19 dist-test-slave-14jq krb5kdc[24032](info): commencing operation
krb5kdc: starting...
W20260927 15:25:20.770860 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:20 dist-test-slave-14jq krb5kdc[24032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:21.081794 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:21.429165 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:21.429486 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:21.429581 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:21.437280 24045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:21.437415 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:21.437461 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:21.437517 24045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:21.437556 24045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:21.448125 24045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:21.449452 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:21.450734 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:21.464162 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:21.464046 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:21.464099 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:21.465519 24045 server_base.cc:1034] running on GCE node
I20260927 15:25:21.466419 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:21.468068 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:21.469333 24045 hybrid_clock.cc:648] HybridClock initialized: now 1790522721469255 us; error 72 us; skew 500 ppm
I20260927 15:25:21.469825 24045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:21 dist-test-slave-14jq krb5kdc[24032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:21.473636 24045 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:21.475621 24045 webserver.cc:460] Webserver started at http://127.0.0.1:33303/ using document root <none> and password file <none>
I20260927 15:25:21.476389 24045 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:21.476542 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:21.476877 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:21.480361 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "977cf45b641e4d51be3ff8bd05198c43"
format_stamp: "Formatted at 2026-09-27 15:25:21 on dist-test-slave-14jq"
server_key: "6e0aaabfcaa79b09c18d8eeec531e79a"
server_key_iv: "97f39120f00fd8ca3ecc18cfbd01007e"
server_key_version: "encryptionkey@0"
I20260927 15:25:21.481191 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "977cf45b641e4d51be3ff8bd05198c43"
format_stamp: "Formatted at 2026-09-27 15:25:21 on dist-test-slave-14jq"
server_key: "6e0aaabfcaa79b09c18d8eeec531e79a"
server_key_iv: "97f39120f00fd8ca3ecc18cfbd01007e"
server_key_version: "encryptionkey@0"
I20260927 15:25:21.487202 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:25:21.491297 24060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:21.493075 24045 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:25:21.493292 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "977cf45b641e4d51be3ff8bd05198c43"
format_stamp: "Formatted at 2026-09-27 15:25:21 on dist-test-slave-14jq"
server_key: "6e0aaabfcaa79b09c18d8eeec531e79a"
server_key_iv: "97f39120f00fd8ca3ecc18cfbd01007e"
server_key_version: "encryptionkey@0"
I20260927 15:25:21.493523 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:21.499892 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:21.504432 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:21.504752 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:21.527153 24045 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39877
I20260927 15:25:21.527174 24111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39877 every 8 connection(s)
I20260927 15:25:21.528582 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:21.533722 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24045
I20260927 15:25:21.533999 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:21.534263 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:21.534559 3033 external_mini_cluster.cc:1428] Setting key 44208095e08db123eba7a4c4ef1bcdb0
I20260927 15:25:21.544062 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:21.536667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47845 (local address 127.0.0.1:39877)
0927 15:25:21.537634 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:21.537648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:21.537700 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:25:21.540103 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:21.540128 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:21.540612 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:21.541122 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:21.542338 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:21.542344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:21.542357 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:21.542376 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:21.542431 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:21.542452 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:21.542716 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:21.542883 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:25:21.543071 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":408,"threads_started":1}
I20260927 15:25:21.549340 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:21.553056 24112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43: Bootstrap starting.
I20260927 15:25:21.557950 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:21.554664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45977 (local address 127.0.0.1:39877)
0927 15:25:21.554938 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:21.554948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:21.555194 (+ 246us) server_negotiation.cc:407] Connection header received
0927 15:25:21.555409 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:21.555417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:21.555536 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:21.555726 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:21.557041 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:21.557046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:21.557051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:21.557073 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:21.557136 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:21.557177 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:21.557334 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:21.557512 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:25:21.557588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:25:21.558638 24112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:21.560277 24112 log.cc:824] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:21.564631 24112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43: No bootstrap required, opened a new log
I20260927 15:25:21.575845 24112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } }
I20260927 15:25:21.576470 24112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:21.576576 24112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977cf45b641e4d51be3ff8bd05198c43, State: Initialized, Role: FOLLOWER
I20260927 15:25:21.577387 24112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } }
I20260927 15:25:21.577804 24112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:21.578030 24112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:21.578341 24112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:21.581463 24112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } }
I20260927 15:25:21.582353 24112 leader_election.cc:302] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977cf45b641e4d51be3ff8bd05198c43; no voters:
I20260927 15:25:21.583648 24112 leader_election.cc:288] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:21.584077 24122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:21.585875 24122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [term 1 LEADER]: Becoming Leader. State: Replica: 977cf45b641e4d51be3ff8bd05198c43, State: Running, Role: LEADER
I20260927 15:25:21.586565 24122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } }
I20260927 15:25:21.587213 24112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:21.590288 24123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977cf45b641e4d51be3ff8bd05198c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } } }
I20260927 15:25:21.590445 24124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977cf45b641e4d51be3ff8bd05198c43. Latest consensus state: current_term: 1 leader_uuid: "977cf45b641e4d51be3ff8bd05198c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977cf45b641e4d51be3ff8bd05198c43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39877 } } }
I20260927 15:25:21.591168 24123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:21.591180 24124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:21.593955 24131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:21.600268 24131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:21.613138 24131 catalog_manager.cc:1347] Generated new cluster ID: a5d610a9d3a04c66be2972c682c8a39f
I20260927 15:25:21.613313 24131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:21.668260 24131 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:21.669019 24131 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:21.683251 24131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977cf45b641e4d51be3ff8bd05198c43: Generated new TSK 0
I20260927 15:25:21.684595 24131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:21.693717 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24045
W20260927 15:25:21.695291 24107 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47845
W20260927 15:25:21.696435 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24045: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:21.696651 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24045
2026-09-27T15:25:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2164 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:21 dist-test-slave-14jq krb5kdc[24143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:21 dist-test-slave-14jq krb5kdc[24143](info): set up 2 sockets
Sep 27 15:25:21 dist-test-slave-14jq krb5kdc[24143](info): commencing operation
krb5kdc: starting...
W20260927 15:25:22.866523 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.007s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:23 dist-test-slave-14jq krb5kdc[24143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:23.192338 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:23.652483 24156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:23.652892 24156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:23.653294 24156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:23.662531 24156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:23.662742 24156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:23.662871 24156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:23.663000 24156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:23.663086 24156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:23.676431 24156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:23.678484 24156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:23.680225 24156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:23.695506 24162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:23.695684 24161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:23.695672 24164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:23.697611 24156 server_base.cc:1034] running on GCE node
I20260927 15:25:23.698813 24156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:23.700742 24156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:23.701987 24156 hybrid_clock.cc:648] HybridClock initialized: now 1790522723701928 us; error 51 us; skew 500 ppm
I20260927 15:25:23.702451 24156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:23 dist-test-slave-14jq krb5kdc[24143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:23.706411 24156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:23.708369 24156 webserver.cc:460] Webserver started at http://127.0.0.1:41765/ using document root <none> and password file <none>
I20260927 15:25:23.709231 24156 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:23.709401 24156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:23.709776 24156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:23.714349 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "5bf5570a386840d3a830df65d3c5e4da"
format_stamp: "Formatted at 2026-09-27 15:25:23 on dist-test-slave-14jq"
server_key: "819de85cc47b31c54adbba92ba850d6f"
server_key_iv: "6439b446eb3fb09db08a77e0e5687050"
server_key_version: "encryptionkey@0"
I20260927 15:25:23.715713 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "5bf5570a386840d3a830df65d3c5e4da"
format_stamp: "Formatted at 2026-09-27 15:25:23 on dist-test-slave-14jq"
server_key: "819de85cc47b31c54adbba92ba850d6f"
server_key_iv: "6439b446eb3fb09db08a77e0e5687050"
server_key_version: "encryptionkey@0"
I20260927 15:25:23.722688 24156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:25:23.727487 24171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:23.729414 24156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:25:23.729615 24156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "5bf5570a386840d3a830df65d3c5e4da"
format_stamp: "Formatted at 2026-09-27 15:25:23 on dist-test-slave-14jq"
server_key: "819de85cc47b31c54adbba92ba850d6f"
server_key_iv: "6439b446eb3fb09db08a77e0e5687050"
server_key_version: "encryptionkey@0"
I20260927 15:25:23.729817 24156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:23.764322 24156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:23.768987 24156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:23.769254 24156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:23.792114 24156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40561
I20260927 15:25:23.792179 24222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40561 every 8 connection(s)
I20260927 15:25:23.793708 24156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:23.798750 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24156
I20260927 15:25:23.799000 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:23.799008 24223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:23.799659 3033 external_mini_cluster.cc:1428] Setting key abb7c276ee511bef60f190b890af2745
Sep 27 15:25:23 dist-test-slave-14jq krb5kdc[24143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:25:23.818387 24223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da: Bootstrap starting.
I20260927 15:25:23.824077 24223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:23.826022 24223 log.cc:824] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:23.830370 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:23.802026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56337 (local address 127.0.0.1:40561)
0927 15:25:23.802983 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:23.802999 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:23.803049 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:25:23.804964 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:23.804990 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:23.805474 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:23.805937 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:23.808414 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:23.809890 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:23.812010 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:23.812871 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:23.813063 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:23.819239 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:23.819258 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:23.819294 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:23.819358 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:23.823851 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:23.825242 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:23.825253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:23.825266 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:23.825563 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:23.826213 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:23.826218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:23.826221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:23.827168 (+ 947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:23.827497 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:23.828261 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 15:25:23.828722 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":402,"threads_started":1}
I20260927 15:25:23.830814 24223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da: No bootstrap required, opened a new log
I20260927 15:25:23.836786 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:23.841943 24223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } }
I20260927 15:25:23.842696 24223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:23.842833 24223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf5570a386840d3a830df65d3c5e4da, State: Initialized, Role: FOLLOWER
I20260927 15:25:23.843868 24223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } }
I20260927 15:25:23.844288 24223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:23.844460 24223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:23.844696 24223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:23.848438 24223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } }
I20260927 15:25:23.849371 24223 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf5570a386840d3a830df65d3c5e4da; no voters:
I20260927 15:25:23.850258 24223 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:23.850455 24233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:23.853359 24223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:23.854156 24233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [term 1 LEADER]: Becoming Leader. State: Replica: 5bf5570a386840d3a830df65d3c5e4da, State: Running, Role: LEADER
I20260927 15:25:23.855562 24233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } }
I20260927 15:25:23.859756 24234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf5570a386840d3a830df65d3c5e4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } } }
I20260927 15:25:23.859663 24235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf5570a386840d3a830df65d3c5e4da. Latest consensus state: current_term: 1 leader_uuid: "5bf5570a386840d3a830df65d3c5e4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf5570a386840d3a830df65d3c5e4da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40561 } } }
I20260927 15:25:23.860311 24234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:23.860407 24235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:23.861716 24240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:23.864280 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:23.842200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:51269 (local address 127.0.0.1:40561)
0927 15:25:23.842453 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:23.842461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:23.842972 (+ 511us) server_negotiation.cc:407] Connection header received
0927 15:25:23.843330 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:23.843338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:23.843468 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:23.843629 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:23.846194 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:23.847131 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:23.849143 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:23.849780 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:23.850021 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:23.854079 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:23.854087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:23.854092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:23.854163 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:23.858827 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:23.860266 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:23.860271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:23.860274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:23.860461 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:23.861186 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:23.861196 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:23.861201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:23.861685 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:23.863219 (+ 1534us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:23.863441 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:25:23.863672 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:25:23.874114 24240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:23.888680 24240 catalog_manager.cc:1347] Generated new cluster ID: 2821d0ea764b417394d03743db9a59ba
I20260927 15:25:23.888852 24240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:23.903772 24240 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:23.905131 24240 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:23.917179 24240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf5570a386840d3a830df65d3c5e4da: Generated new TSK 0
I20260927 15:25:23.918928 24240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:23.935397 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24156
I20260927 15:25:23.936733 24218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:24.047448 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24156
W20260927 15:25:24.066913 24141 connection.cc:535] client connection to 127.0.0.1:40561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40561): Connection reset by peer (error 104)
2026-09-27T15:25:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:24 dist-test-slave-14jq krb5kdc[24255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:24 dist-test-slave-14jq krb5kdc[24255](info): set up 2 sockets
Sep 27 15:25:24 dist-test-slave-14jq krb5kdc[24255](info): commencing operation
krb5kdc: starting...
W20260927 15:25:25.210637 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:25 dist-test-slave-14jq krb5kdc[24255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:25.526750 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:25.895604 24268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:25.895929 24268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:25.896076 24268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:25.903671 24268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:25.903822 24268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:25.903911 24268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:25.904017 24268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:25.904111 24268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:25.914381 24268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:25.915784 24268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:25.917232 24268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:25.932366 24273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:25.932708 24274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:25.932464 24276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:25.934573 24268 server_base.cc:1034] running on GCE node
I20260927 15:25:25.935544 24268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:25.937477 24268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:25.938872 24268 hybrid_clock.cc:648] HybridClock initialized: now 1790522725938777 us; error 103 us; skew 500 ppm
I20260927 15:25:25.939438 24268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:25 dist-test-slave-14jq krb5kdc[24255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:25.948246 24268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:25.950706 24268 webserver.cc:460] Webserver started at http://127.0.0.1:36995/ using document root <none> and password file <none>
I20260927 15:25:25.951678 24268 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:25.951810 24268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:25.952127 24268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:25.956173 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "0fd44686fe0a4d1eb338954287dcabbf"
format_stamp: "Formatted at 2026-09-27 15:25:25 on dist-test-slave-14jq"
server_key: "7e78f940cfccb46bd6ea7812d75c7510"
server_key_iv: "2f9acb3ff1e7102a0782d38c82d71e61"
server_key_version: "encryptionkey@0"
I20260927 15:25:25.957170 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "0fd44686fe0a4d1eb338954287dcabbf"
format_stamp: "Formatted at 2026-09-27 15:25:25 on dist-test-slave-14jq"
server_key: "7e78f940cfccb46bd6ea7812d75c7510"
server_key_iv: "2f9acb3ff1e7102a0782d38c82d71e61"
server_key_version: "encryptionkey@0"
I20260927 15:25:25.964550 24268 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:25:25.969987 24283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:25.972072 24268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:25:25.972378 24268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "0fd44686fe0a4d1eb338954287dcabbf"
format_stamp: "Formatted at 2026-09-27 15:25:25 on dist-test-slave-14jq"
server_key: "7e78f940cfccb46bd6ea7812d75c7510"
server_key_iv: "2f9acb3ff1e7102a0782d38c82d71e61"
server_key_version: "encryptionkey@0"
I20260927 15:25:25.972688 24268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:26.004854 24268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:26.010136 24268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:26.010581 24268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:26.037213 24268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43943
I20260927 15:25:26.037264 24334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43943 every 8 connection(s)
I20260927 15:25:26.039166 24268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:26.041602 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24268
I20260927 15:25:26.041821 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:26.042470 3033 external_mini_cluster.cc:1428] Setting key 5452d36ae5e69e41fcc05238fd765f3a
I20260927 15:25:26.044570 24335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:26.060001 24335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf: Bootstrap starting.
I20260927 15:25:26.060617 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:26.045082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56837 (local address 127.0.0.1:43943)
0927 15:25:26.046248 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:26.046265 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:26.046324 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:25:26.048315 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:26.048356 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:26.048971 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:26.049485 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:26.050212 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:26.052078 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:26.054234 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:26.055175 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:26.055385 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:26.057069 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:26.057081 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:26.057111 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:26.058071 (+ 960us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:26.058201 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:26.058237 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:26.058416 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:26.058806 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:25:26.059131 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":518,"threads_started":1}
I20260927 15:25:26.063654 24335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:26.065032 24335 log.cc:824] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:26.066179 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:26.068194 24335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf: No bootstrap required, opened a new log
I20260927 15:25:26.078393 24335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260927 15:25:26.078977 24335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:26.079113 24335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd44686fe0a4d1eb338954287dcabbf, State: Initialized, Role: FOLLOWER
I20260927 15:25:26.079998 24335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260927 15:25:26.080307 24335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:26.080523 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:26.071746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:38021 (local address 127.0.0.1:43943)
0927 15:25:26.072080 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:26.072092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:26.072549 (+ 457us) server_negotiation.cc:407] Connection header received
0927 15:25:26.072835 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:26.072841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:26.072934 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:26.073065 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:26.073781 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:26.074799 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:26.076983 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:26.077616 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:26.077745 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:26.079433 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:26.079442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:26.079447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:26.079473 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:26.079575 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:26.079604 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:26.079750 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:26.080056 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:25:26.080130 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 15:25:26.080519 24335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:26.080749 24335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:26.083765 24335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260927 15:25:26.084396 24335 leader_election.cc:302] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd44686fe0a4d1eb338954287dcabbf; no voters:
I20260927 15:25:26.085202 24335 leader_election.cc:288] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:26.085563 24345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:26.087239 24345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [term 1 LEADER]: Becoming Leader. State: Replica: 0fd44686fe0a4d1eb338954287dcabbf, State: Running, Role: LEADER
I20260927 15:25:26.088019 24335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:26.087991 24345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260927 15:25:26.091930 24347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd44686fe0a4d1eb338954287dcabbf. Latest consensus state: current_term: 1 leader_uuid: "0fd44686fe0a4d1eb338954287dcabbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260927 15:25:26.092512 24347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:26.093788 24346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fd44686fe0a4d1eb338954287dcabbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd44686fe0a4d1eb338954287dcabbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260927 15:25:26.095009 24346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:26.101446 24356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:26.107056 24356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:26.117426 24356 catalog_manager.cc:1347] Generated new cluster ID: 85d9d428165640b1b7f1f8b8c789b653
I20260927 15:25:26.117591 24356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:26.133134 24356 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:26.135687 24356 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:26.149482 24356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fd44686fe0a4d1eb338954287dcabbf: Generated new TSK 0
I20260927 15:25:26.151333 24356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:26.167069 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24268
W20260927 15:25:26.169971 24330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:56837
W20260927 15:25:26.171271 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24268: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:26.171545 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24268
W20260927 15:25:26.194085 24253 connection.cc:535] client connection to 127.0.0.1:43943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43943): Connection reset by peer (error 104)
2026-09-27T15:25:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2128 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:26 dist-test-slave-14jq krb5kdc[24366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:26 dist-test-slave-14jq krb5kdc[24366](info): set up 2 sockets
Sep 27 15:25:26 dist-test-slave-14jq krb5kdc[24366](info): commencing operation
krb5kdc: starting...
W20260927 15:25:27.338965 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:27 dist-test-slave-14jq krb5kdc[24366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:27.650089 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:27.998020 24379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:27.998345 24379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:27.998504 24379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:28.006175 24379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:28.006316 24379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:28.006363 24379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:28.006418 24379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:28.006526 24379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:28.017231 24379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:28.018703 24379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:28.020126 24379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:28.033556 24387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:28.033627 24385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:28.033689 24384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:28.034795 24379 server_base.cc:1034] running on GCE node
I20260927 15:25:28.035825 24379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:28.037472 24379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:28.038761 24379 hybrid_clock.cc:648] HybridClock initialized: now 1790522728038699 us; error 77 us; skew 500 ppm
I20260927 15:25:28.039193 24379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:28 dist-test-slave-14jq krb5kdc[24366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:28.042870 24379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:28.044767 24379 webserver.cc:460] Webserver started at http://127.0.0.1:34361/ using document root <none> and password file <none>
I20260927 15:25:28.045584 24379 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:28.045744 24379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:28.046144 24379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:28.049326 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "37d0d30a3ffc400db0fa04623d128db8"
format_stamp: "Formatted at 2026-09-27 15:25:28 on dist-test-slave-14jq"
server_key: "8cc73a93ea5c692bb71ba06fc2334f8b"
server_key_iv: "0b695763e5283b8b0056fc0a60a59c0f"
server_key_version: "encryptionkey@0"
I20260927 15:25:28.050220 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "37d0d30a3ffc400db0fa04623d128db8"
format_stamp: "Formatted at 2026-09-27 15:25:28 on dist-test-slave-14jq"
server_key: "8cc73a93ea5c692bb71ba06fc2334f8b"
server_key_iv: "0b695763e5283b8b0056fc0a60a59c0f"
server_key_version: "encryptionkey@0"
I20260927 15:25:28.056548 24379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:25:28.060878 24394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:28.062896 24379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:25:28.063084 24379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "37d0d30a3ffc400db0fa04623d128db8"
format_stamp: "Formatted at 2026-09-27 15:25:28 on dist-test-slave-14jq"
server_key: "8cc73a93ea5c692bb71ba06fc2334f8b"
server_key_iv: "0b695763e5283b8b0056fc0a60a59c0f"
server_key_version: "encryptionkey@0"
I20260927 15:25:28.063320 24379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:28.070494 24379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:28.076319 24379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:28.076644 24379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:28.101738 24379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43523
I20260927 15:25:28.101775 24445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43523 every 8 connection(s)
I20260927 15:25:28.104185 24379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:28.108518 24446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:28.113054 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24379
I20260927 15:25:28.113286 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:28.113914 3033 external_mini_cluster.cc:1428] Setting key a6ed10b9c07643019d318a45e81965a1
I20260927 15:25:28.120368 24446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8: Bootstrap starting.
I20260927 15:25:28.124316 24446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:28.124471 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:28.116433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:53485 (local address 127.0.0.1:43523)
0927 15:25:28.117542 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:28.117560 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:28.117631 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:25:28.119830 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:28.119863 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:28.120365 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:28.120923 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:28.122221 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:28.122229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:28.122246 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:28.122265 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:28.122342 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:28.122364 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:28.122582 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:28.122698 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 15:25:28.123010 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":521,"threads_started":1}
I20260927 15:25:28.125597 24446 log.cc:824] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:28.129282 24446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8: No bootstrap required, opened a new log
I20260927 15:25:28.129793 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:28.138188 24446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } }
I20260927 15:25:28.138760 24446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:28.138893 24446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d0d30a3ffc400db0fa04623d128db8, State: Initialized, Role: FOLLOWER
I20260927 15:25:28.139110 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:28.135360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:39541 (local address 127.0.0.1:43523)
0927 15:25:28.135669 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:28.135681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:28.136260 (+ 579us) server_negotiation.cc:407] Connection header received
0927 15:25:28.136519 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:28.136524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:28.136590 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:28.136736 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:28.138243 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:28.138248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:28.138252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:28.138271 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:28.138324 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:28.138341 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:28.138467 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:28.138592 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 15:25:28.138689 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:25:28.140209 24446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } }
I20260927 15:25:28.140539 24446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:28.140727 24446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:28.140920 24446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:28.143383 24446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } }
I20260927 15:25:28.144038 24446 leader_election.cc:302] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d0d30a3ffc400db0fa04623d128db8; no voters:
I20260927 15:25:28.144801 24446 leader_election.cc:288] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:28.145193 24456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:28.146310 24456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [term 1 LEADER]: Becoming Leader. State: Replica: 37d0d30a3ffc400db0fa04623d128db8, State: Running, Role: LEADER
I20260927 15:25:28.146993 24456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } }
I20260927 15:25:28.147437 24446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:28.150933 24457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d0d30a3ffc400db0fa04623d128db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } } }
I20260927 15:25:28.151504 24457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:28.150892 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d0d30a3ffc400db0fa04623d128db8. Latest consensus state: current_term: 1 leader_uuid: "37d0d30a3ffc400db0fa04623d128db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0d30a3ffc400db0fa04623d128db8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43523 } } }
I20260927 15:25:28.152241 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:28.158841 24465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:28.165347 24465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:28.176821 24465 catalog_manager.cc:1347] Generated new cluster ID: c0bb424c6a0349f5bea44e112f4ad16c
I20260927 15:25:28.177026 24465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:28.191094 24465 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:28.191581 24465 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:28.204497 24465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d0d30a3ffc400db0fa04623d128db8: Generated new TSK 0
I20260927 15:25:28.205457 24465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:28.214596 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24379
W20260927 15:25:28.216182 24441 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:53485
W20260927 15:25:28.217234 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24379: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:28.217454 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24379
2026-09-27T15:25:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:28 dist-test-slave-14jq krb5kdc[24477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:28 dist-test-slave-14jq krb5kdc[24477](info): set up 2 sockets
Sep 27 15:25:28 dist-test-slave-14jq krb5kdc[24477](info): commencing operation
krb5kdc: starting...
W20260927 15:25:29.362812 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:29 dist-test-slave-14jq krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:29.698372 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:30.085827 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:30.086231 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:30.086416 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:30.094427 24490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:30.094574 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:30.094623 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:30.094754 24490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:30.094836 24490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:30.105584 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:30.107164 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:30.108645 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:30.123456 24498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:30.123554 24496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:30.123554 24495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:30.124418 24490 server_base.cc:1034] running on GCE node
I20260927 15:25:30.125689 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:30.128321 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:30.129706 24490 hybrid_clock.cc:648] HybridClock initialized: now 1790522730129614 us; error 84 us; skew 500 ppm
I20260927 15:25:30.130353 24490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:30 dist-test-slave-14jq krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:30.135715 24490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:30.137812 24490 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
I20260927 15:25:30.139356 24490 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:30.139591 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:30.140013 24490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:30.144315 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "3cec0a5c8f8549ceb608740aff9a3df0"
format_stamp: "Formatted at 2026-09-27 15:25:30 on dist-test-slave-14jq"
server_key: "690f263287456ec28b8657a33a20e52d"
server_key_iv: "4becdce792a0c6df773c7a37780829da"
server_key_version: "encryptionkey@0"
I20260927 15:25:30.145431 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "3cec0a5c8f8549ceb608740aff9a3df0"
format_stamp: "Formatted at 2026-09-27 15:25:30 on dist-test-slave-14jq"
server_key: "690f263287456ec28b8657a33a20e52d"
server_key_iv: "4becdce792a0c6df773c7a37780829da"
server_key_version: "encryptionkey@0"
I20260927 15:25:30.153642 24490 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.004s
I20260927 15:25:30.159596 24505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:30.161821 24490 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:25:30.162250 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "3cec0a5c8f8549ceb608740aff9a3df0"
format_stamp: "Formatted at 2026-09-27 15:25:30 on dist-test-slave-14jq"
server_key: "690f263287456ec28b8657a33a20e52d"
server_key_iv: "4becdce792a0c6df773c7a37780829da"
server_key_version: "encryptionkey@0"
I20260927 15:25:30.162562 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:30.188642 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:30.195999 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:30.196440 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:30.221864 24490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41873
I20260927 15:25:30.221940 24556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41873 every 8 connection(s)
I20260927 15:25:30.223730 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:30.228302 24557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:30.233657 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24490
I20260927 15:25:30.233989 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:30.235052 3033 external_mini_cluster.cc:1428] Setting key 43250c18ad6f44e8a1ac7d89100acf07
I20260927 15:25:30.240463 24557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0: Bootstrap starting.
I20260927 15:25:30.244100 24557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:30.245659 24557 log.cc:824] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:30.248978 24557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0: No bootstrap required, opened a new log
Sep 27 15:25:30 dist-test-slave-14jq krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:25:30.258288 24557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260927 15:25:30.258797 24557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:30.258957 24557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cec0a5c8f8549ceb608740aff9a3df0, State: Initialized, Role: FOLLOWER
I20260927 15:25:30.259658 24557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260927 15:25:30.259963 24557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:30.260149 24557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:30.260555 24557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:30.263859 24557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260927 15:25:30.264551 24557 leader_election.cc:302] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cec0a5c8f8549ceb608740aff9a3df0; no voters:
I20260927 15:25:30.265443 24557 leader_election.cc:288] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:30.265689 24562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:30.267910 24557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:30.269085 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:30.237067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:33497 (local address 127.0.0.1:41873)
0927 15:25:30.238228 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:30.238246 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:30.238312 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:25:30.240455 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:30.240488 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:30.241164 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:30.241710 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:30.243588 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:30.245608 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:30.247196 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:30.248170 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:30.248532 (+ 362us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:30.255173 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:30.255190 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:30.255221 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:30.255304 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:30.260473 (+ 5169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:30.262104 (+ 1631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:30.262118 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:30.262136 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:30.262427 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:30.263066 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:30.263070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:30.263072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:30.264195 (+ 1123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:30.264553 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:30.265767 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0927 15:25:30.266428 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":436,"threads_started":1}
I20260927 15:25:30.270160 24562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [term 1 LEADER]: Becoming Leader. State: Replica: 3cec0a5c8f8549ceb608740aff9a3df0, State: Running, Role: LEADER
I20260927 15:25:30.271250 24562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260927 15:25:30.276413 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:30.284492 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260927 15:25:30.285066 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:30.285802 24564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cec0a5c8f8549ceb608740aff9a3df0. Latest consensus state: current_term: 1 leader_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cec0a5c8f8549ceb608740aff9a3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260927 15:25:30.286226 24564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:30.286543 24575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:30.303390 24575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:30.320969 24575 catalog_manager.cc:1347] Generated new cluster ID: 857ab4dbbca441b396aee9f0f8b160e1
I20260927 15:25:30.321120 24575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:30.322657 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:30.297826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56403 (local address 127.0.0.1:41873)
0927 15:25:30.298092 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:30.298099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:30.302194 (+ 4095us) server_negotiation.cc:407] Connection header received
0927 15:25:30.302487 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:30.302494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:30.302592 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:30.302739 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:30.305208 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:30.306731 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:30.308717 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:30.309419 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:30.309554 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:30.313606 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:30.313614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:30.313618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:30.313668 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:30.318016 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:30.319317 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:30.319323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:30.319326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:30.319506 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:30.320065 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:30.320070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:30.320073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:30.320537 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:30.320866 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:30.321945 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0927 15:25:30.322214 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:25:30.351290 24575 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:30.353085 24575 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:30.368211 24575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cec0a5c8f8549ceb608740aff9a3df0: Generated new TSK 0
I20260927 15:25:30.369627 24575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:30.383878 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24490
I20260927 15:25:30.385736 24552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:30.485217 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24490
W20260927 15:25:30.503352 24475 connection.cc:535] client connection to 127.0.0.1:41873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41873): Connection reset by peer (error 104)
2026-09-27T15:25:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:30 dist-test-slave-14jq krb5kdc[24589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:30 dist-test-slave-14jq krb5kdc[24589](info): set up 2 sockets
Sep 27 15:25:30 dist-test-slave-14jq krb5kdc[24589](info): commencing operation
krb5kdc: starting...
W20260927 15:25:31.670499 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:31 dist-test-slave-14jq krb5kdc[24589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:31.985808 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:32.429916 24602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:32.430425 24602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:32.430626 24602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:32.439407 24602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:32.439635 24602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:32.439776 24602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:32.439906 24602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:32.440006 24602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:32.452351 24602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:32.454231 24602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:32.455930 24602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:32.470341 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:32.470460 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:32.470286 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:32.472769 24602 server_base.cc:1034] running on GCE node
I20260927 15:25:32.473975 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:32.475971 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:32.477290 24602 hybrid_clock.cc:648] HybridClock initialized: now 1790522732477226 us; error 55 us; skew 500 ppm
I20260927 15:25:32.477802 24602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:32 dist-test-slave-14jq krb5kdc[24589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:32.488109 24602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:32.490401 24602 webserver.cc:460] Webserver started at http://127.0.0.1:45853/ using document root <none> and password file <none>
I20260927 15:25:32.491364 24602 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:32.491531 24602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:32.491914 24602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:32.495818 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "5be4e56132cf468fafb2ddf279f8b714"
format_stamp: "Formatted at 2026-09-27 15:25:32 on dist-test-slave-14jq"
server_key: "793a0f40f42a1ac1489435410a1aeb37"
server_key_iv: "c27aa8f2e1dbf71bdad3814d6fc23df8"
server_key_version: "encryptionkey@0"
I20260927 15:25:32.496833 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "5be4e56132cf468fafb2ddf279f8b714"
format_stamp: "Formatted at 2026-09-27 15:25:32 on dist-test-slave-14jq"
server_key: "793a0f40f42a1ac1489435410a1aeb37"
server_key_iv: "c27aa8f2e1dbf71bdad3814d6fc23df8"
server_key_version: "encryptionkey@0"
I20260927 15:25:32.504211 24602 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260927 15:25:32.509516 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:32.511390 24602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:25:32.511641 24602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "5be4e56132cf468fafb2ddf279f8b714"
format_stamp: "Formatted at 2026-09-27 15:25:32 on dist-test-slave-14jq"
server_key: "793a0f40f42a1ac1489435410a1aeb37"
server_key_iv: "c27aa8f2e1dbf71bdad3814d6fc23df8"
server_key_version: "encryptionkey@0"
I20260927 15:25:32.511901 24602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:32.535764 24602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:32.541342 24602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:32.541706 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:32.566987 24602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34997
I20260927 15:25:32.566926 24668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34997 every 8 connection(s)
I20260927 15:25:32.568919 24602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:32.570902 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24602
I20260927 15:25:32.571135 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:32.571743 3033 external_mini_cluster.cc:1428] Setting key 5310256ade0030eb62be1f6b2030c11d
I20260927 15:25:32.575042 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:32.589618 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:32.574281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:36059 (local address 127.0.0.1:34997)
0927 15:25:32.575397 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:32.575413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:32.575470 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:25:32.577488 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:32.577518 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:32.578202 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:32.578694 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:32.579482 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:32.581138 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:32.583386 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:32.584393 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:32.584604 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:32.586374 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:32.586385 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:32.586406 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:32.587220 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:32.587341 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:32.587388 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:32.587570 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:32.587991 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:25:32.588293 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":506,"threads_started":1}
I20260927 15:25:32.590740 24669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714: Bootstrap starting.
I20260927 15:25:32.594893 24669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:32.595676 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:32.596237 24669 log.cc:824] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:32.600273 24669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714: No bootstrap required, opened a new log
I20260927 15:25:32.608415 24669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } }
I20260927 15:25:32.608872 24669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:32.609045 24669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be4e56132cf468fafb2ddf279f8b714, State: Initialized, Role: FOLLOWER
I20260927 15:25:32.609815 24669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } }
I20260927 15:25:32.610152 24669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:32.610329 24669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:32.610530 24669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:32.613063 24669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } }
I20260927 15:25:32.613663 24669 leader_election.cc:302] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5be4e56132cf468fafb2ddf279f8b714; no voters:
I20260927 15:25:32.614464 24669 leader_election.cc:288] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:32.614614 24679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:32.615389 24679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [term 1 LEADER]: Becoming Leader. State: Replica: 5be4e56132cf468fafb2ddf279f8b714, State: Running, Role: LEADER
I20260927 15:25:32.616183 24679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } }
I20260927 15:25:32.617298 24669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:32.617301 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:32.602417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:37569 (local address 127.0.0.1:34997)
0927 15:25:32.602664 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:32.602671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:32.609587 (+ 6916us) server_negotiation.cc:407] Connection header received
0927 15:25:32.609996 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:32.610001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:32.610127 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:32.610306 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:32.611039 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:32.612022 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:32.613525 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:32.614195 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:32.614349 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:32.616014 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:32.616022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:32.616027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:32.616058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:32.616121 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:32.616152 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:32.616296 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:32.616817 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:25:32.616886 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:25:32.619977 24681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5be4e56132cf468fafb2ddf279f8b714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } } }
I20260927 15:25:32.620539 24681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:32.621681 24680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be4e56132cf468fafb2ddf279f8b714. Latest consensus state: current_term: 1 leader_uuid: "5be4e56132cf468fafb2ddf279f8b714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be4e56132cf468fafb2ddf279f8b714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34997 } } }
I20260927 15:25:32.622661 24680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:32.623991 24684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:32.635073 24684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:32.647040 24684 catalog_manager.cc:1347] Generated new cluster ID: e9c6cd1ca1f243269c761917caf9a67a
I20260927 15:25:32.647213 24684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:32.684890 24684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:32.686902 24684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:32.694291 24684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5be4e56132cf468fafb2ddf279f8b714: Generated new TSK 0
I20260927 15:25:32.695464 24684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:32.703385 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24602
W20260927 15:25:32.705433 24664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:36059
W20260927 15:25:32.706431 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:32.706597 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24602
W20260927 15:25:32.725144 24587 connection.cc:535] client connection to 127.0.0.1:34997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34997): Connection reset by peer (error 104)
2026-09-27T15:25:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:32 dist-test-slave-14jq krb5kdc[24701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:32 dist-test-slave-14jq krb5kdc[24701](info): set up 2 sockets
Sep 27 15:25:32 dist-test-slave-14jq krb5kdc[24701](info): commencing operation
krb5kdc: starting...
W20260927 15:25:33.890980 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:34 dist-test-slave-14jq krb5kdc[24701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:25:34.205546 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:34.582587 24714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:34.582978 24714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:34.583146 24714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:34.591171 24714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:34.591405 24714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:34.591526 24714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:34.591645 24714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:34.591745 24714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:34.602723 24714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:34.604312 24714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:34.605641 24714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:34.620783 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:34.620978 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:34.621954 24714 server_base.cc:1034] running on GCE node
W20260927 15:25:34.622103 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:34.623219 24714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:34.624890 24714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:34.626183 24714 hybrid_clock.cc:648] HybridClock initialized: now 1790522734626120 us; error 59 us; skew 500 ppm
I20260927 15:25:34.626632 24714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:34 dist-test-slave-14jq krb5kdc[24701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:34.631660 24714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:25:34.634243 24714 webserver.cc:460] Webserver started at http://127.0.0.1:39015/ using document root <none> and password file <none>
I20260927 15:25:34.635295 24714 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:34.635486 24714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:34.635881 24714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:34.639856 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "14de1eba631a4e3e8d1e9e940761a487"
format_stamp: "Formatted at 2026-09-27 15:25:34 on dist-test-slave-14jq"
server_key: "2d50b76d46c836cd20d6e7cc9de800fa"
server_key_iv: "d4e4807bfb3412673bed6338492a4450"
server_key_version: "encryptionkey@0"
I20260927 15:25:34.641111 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "14de1eba631a4e3e8d1e9e940761a487"
format_stamp: "Formatted at 2026-09-27 15:25:34 on dist-test-slave-14jq"
server_key: "2d50b76d46c836cd20d6e7cc9de800fa"
server_key_iv: "d4e4807bfb3412673bed6338492a4450"
server_key_version: "encryptionkey@0"
I20260927 15:25:34.648882 24714 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:25:34.654160 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:34.656260 24714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:25:34.656581 24714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "14de1eba631a4e3e8d1e9e940761a487"
format_stamp: "Formatted at 2026-09-27 15:25:34 on dist-test-slave-14jq"
server_key: "2d50b76d46c836cd20d6e7cc9de800fa"
server_key_iv: "d4e4807bfb3412673bed6338492a4450"
server_key_version: "encryptionkey@0"
I20260927 15:25:34.656893 24714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:34.665666 24714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:34.672688 24714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:34.673153 24714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:34.696750 24714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33269
I20260927 15:25:34.696805 24780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33269 every 8 connection(s)
I20260927 15:25:34.698711 24714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:34.699635 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24714
I20260927 15:25:34.699846 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:34.700392 3033 external_mini_cluster.cc:1428] Setting key 077a9d476ce21ce70afccde6b7c22ad0
I20260927 15:25:34.705307 24781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:34.710070 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:34.702732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:46429 (local address 127.0.0.1:33269)
0927 15:25:34.703852 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:34.703905 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:34.703977 (+ 72us) server_negotiation.cc:407] Connection header received
0927 15:25:34.706096 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:34.706124 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:34.706558 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:34.707012 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:34.708172 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:34.708179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:34.708193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:34.708209 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:34.708287 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:34.708308 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:34.708582 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:34.708743 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:25:34.708958 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":510,"threads_started":1}
I20260927 15:25:34.715234 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:34.721437 24781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487: Bootstrap starting.
I20260927 15:25:34.722932 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:34.720217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:58213 (local address 127.0.0.1:33269)
0927 15:25:34.720422 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:34.720429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:34.720826 (+ 397us) server_negotiation.cc:407] Connection header received
0927 15:25:34.721044 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:34.721048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:34.721098 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:34.721208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:34.722232 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:34.722238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:34.722242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:34.722260 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:34.722308 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:34.722326 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:34.722453 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:34.722583 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 15:25:34.722653 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:25:34.726233 24781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:34.727563 24781 log.cc:824] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:34.731706 24781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487: No bootstrap required, opened a new log
I20260927 15:25:34.740880 24781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260927 15:25:34.741430 24781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:34.741549 24781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14de1eba631a4e3e8d1e9e940761a487, State: Initialized, Role: FOLLOWER
I20260927 15:25:34.742532 24781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260927 15:25:34.743145 24781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:34.743356 24781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:34.743577 24781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:34.746742 24781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260927 15:25:34.747591 24781 leader_election.cc:302] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14de1eba631a4e3e8d1e9e940761a487; no voters:
I20260927 15:25:34.748503 24781 leader_election.cc:288] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:34.748734 24791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:34.750387 24791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [term 1 LEADER]: Becoming Leader. State: Replica: 14de1eba631a4e3e8d1e9e940761a487, State: Running, Role: LEADER
I20260927 15:25:34.751221 24791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260927 15:25:34.751576 24781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:34.755640 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14de1eba631a4e3e8d1e9e940761a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260927 15:25:34.755118 24793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14de1eba631a4e3e8d1e9e940761a487. Latest consensus state: current_term: 1 leader_uuid: "14de1eba631a4e3e8d1e9e940761a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de1eba631a4e3e8d1e9e940761a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260927 15:25:34.756345 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:34.756394 24793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:34.759085 24800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:34.765007 24800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:34.778477 24800 catalog_manager.cc:1347] Generated new cluster ID: db47272c129d4cf38807223636d7978d
I20260927 15:25:34.778632 24800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:34.805316 24800 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:34.806136 24800 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:34.818130 24800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14de1eba631a4e3e8d1e9e940761a487: Generated new TSK 0
I20260927 15:25:34.819473 24800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:34.842383 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24714
W20260927 15:25:34.843909 24776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:46429
W20260927 15:25:34.844960 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24714: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:34.845172 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24714
2026-09-27T15:25:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:34 dist-test-slave-14jq krb5kdc[24812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:34 dist-test-slave-14jq krb5kdc[24812](info): set up 2 sockets
Sep 27 15:25:34 dist-test-slave-14jq krb5kdc[24812](info): commencing operation
krb5kdc: starting...
W20260927 15:25:36.010775 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:36 dist-test-slave-14jq krb5kdc[24812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:36.326081 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34057
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:34057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:36.756981 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:36.757541 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:36.757663 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:36.765990 24825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:36.766170 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:36.766227 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:36.766289 24825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:36.766345 24825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:36.778159 24825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:34057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:36.780373 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:36.782450 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:36.796948 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:36.797029 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:36.797008 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:36.798772 24825 server_base.cc:1034] running on GCE node
I20260927 15:25:36.799829 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:36.801616 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:36.802925 24825 hybrid_clock.cc:648] HybridClock initialized: now 1790522736802867 us; error 50 us; skew 500 ppm
I20260927 15:25:36.803391 24825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:36 dist-test-slave-14jq krb5kdc[24812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522736, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:36.807183 24825 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:36.809231 24825 webserver.cc:460] Webserver started at http://127.2.246.126:43739/ using document root <none> and password file <none>
I20260927 15:25:36.810236 24825 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:36.810403 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:36.811175 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:36.816205 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "320b8d4a8bcb47419a7d04b96939ebd2"
format_stamp: "Formatted at 2026-09-27 15:25:36 on dist-test-slave-14jq"
server_key: "e9676f0d955b0ab009ce58f4917cb76e"
server_key_iv: "9b1a9fca77bd54d0a0cff056444e85e1"
server_key_version: "encryptionkey@0"
I20260927 15:25:36.817704 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "320b8d4a8bcb47419a7d04b96939ebd2"
format_stamp: "Formatted at 2026-09-27 15:25:36 on dist-test-slave-14jq"
server_key: "e9676f0d955b0ab009ce58f4917cb76e"
server_key_iv: "9b1a9fca77bd54d0a0cff056444e85e1"
server_key_version: "encryptionkey@0"
I20260927 15:25:36.826367 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 15:25:36.832547 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:36.834914 24825 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:25:36.835237 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "320b8d4a8bcb47419a7d04b96939ebd2"
format_stamp: "Formatted at 2026-09-27 15:25:36 on dist-test-slave-14jq"
server_key: "e9676f0d955b0ab009ce58f4917cb76e"
server_key_iv: "9b1a9fca77bd54d0a0cff056444e85e1"
server_key_version: "encryptionkey@0"
I20260927 15:25:36.835543 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:36.861459 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:36.868067 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:36.868503 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:36.894208 24825 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:34057
I20260927 15:25:36.894236 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:34057 every 8 connection(s)
I20260927 15:25:36.896020 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:36.900497 24892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:36.901126 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24825
I20260927 15:25:36.901355 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:36.902146 3033 external_mini_cluster.cc:1428] Setting key c34d4527bf71209a23e472debb569d44
I20260927 15:25:36.915733 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2: Bootstrap starting.
I20260927 15:25:36.919353 24892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2: Neither blocks nor log segments found. Creating new log.
Sep 27 15:25:36 dist-test-slave-14jq krb5kdc[24812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:36.920567 24892 log.cc:824] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:36.924777 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2: No bootstrap required, opened a new log
I20260927 15:25:36.932708 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:36.904836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54803 (local address 127.2.246.126:34057)
0927 15:25:36.906017 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:36.906040 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:36.906115 (+ 75us) server_negotiation.cc:407] Connection header received
0927 15:25:36.907902 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:36.907934 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:36.908550 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:36.909037 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:36.911033 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:36.913120 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:36.915388 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:36.916439 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:36.916632 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:36.922353 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:36.922362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:36.922379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:36.922483 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:36.926827 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:36.928065 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:36.928075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:36.928084 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:36.928349 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:36.928986 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:36.928991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:36.928995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:36.929813 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:36.930242 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:36.930799 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 15:25:36.931214 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":521,"threads_started":1}
I20260927 15:25:36.935532 24892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } }
I20260927 15:25:36.936028 24892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:36.936126 24892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320b8d4a8bcb47419a7d04b96939ebd2, State: Initialized, Role: FOLLOWER
I20260927 15:25:36.936686 24892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } }
I20260927 15:25:36.936946 24892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:36.937098 24892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:36.937315 24892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:36.938198 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:36.940631 24892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } }
I20260927 15:25:36.941304 24892 leader_election.cc:302] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320b8d4a8bcb47419a7d04b96939ebd2; no voters:
I20260927 15:25:36.942337 24892 leader_election.cc:288] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:36.942538 24898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:36.943362 24898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [term 1 LEADER]: Becoming Leader. State: Replica: 320b8d4a8bcb47419a7d04b96939ebd2, State: Running, Role: LEADER
I20260927 15:25:36.944187 24898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } }
I20260927 15:25:36.947909 24902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } } }
I20260927 15:25:36.948478 24902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:36.948908 24892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:36.954913 24901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320b8d4a8bcb47419a7d04b96939ebd2. Latest consensus state: current_term: 1 leader_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320b8d4a8bcb47419a7d04b96939ebd2" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 34057 } } }
I20260927 15:25:36.955672 24906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:36.955678 24901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:36.963225 24906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:36.978763 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:36.949127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34389 (local address 127.2.246.126:34057)
0927 15:25:36.949432 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:36.949440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:36.953709 (+ 4269us) server_negotiation.cc:407] Connection header received
0927 15:25:36.954864 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:36.954873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:36.955015 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:36.955156 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:36.957772 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:36.959604 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:36.962257 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:36.963320 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:36.963494 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:36.968102 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:36.968117 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:36.968126 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:36.968189 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:36.973343 (+ 5154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:36.974900 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:36.974906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:36.974913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:36.975092 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:36.975744 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:36.975749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:36.975752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:36.976319 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:36.976690 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:36.978041 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0927 15:25:36.978289 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 15:25:36.980881 24906 catalog_manager.cc:1347] Generated new cluster ID: 402b4ad263de4e1ab413ae2cc03c9e6f
I20260927 15:25:36.981098 24906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:36.993522 24906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:36.994961 24906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:37.003837 24906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320b8d4a8bcb47419a7d04b96939ebd2: Generated new TSK 0
I20260927 15:25:37.005378 24906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:37.017419 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24825
I20260927 15:25:37.019945 24887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:37.126021 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24825
W20260927 15:25:37.146076 24810 connection.cc:535] client connection to 127.2.246.126:34057 recv error: Network error: failed to read from TLS socket (remote: 127.2.246.126:34057): Connection reset by peer (error 104)
2026-09-27T15:25:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:37 dist-test-slave-14jq krb5kdc[24924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:37 dist-test-slave-14jq krb5kdc[24924](info): set up 2 sockets
Sep 27 15:25:37 dist-test-slave-14jq krb5kdc[24924](info): commencing operation
krb5kdc: starting...
W20260927 15:25:38.283648 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.007s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:38 dist-test-slave-14jq krb5kdc[24924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:38.598213 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44213
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:44213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:38.968932 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:38.969282 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:38.969383 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:38.976905 24937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:38.977062 24937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:38.977115 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:38.977171 24937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:38.977211 24937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:38.987522 24937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:44213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:38.988909 24937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:38.990314 24937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:39.006295 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:39.006379 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:39.006460 24945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:39.008697 24937 server_base.cc:1034] running on GCE node
I20260927 15:25:39.009727 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:39.011618 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:39.013000 24937 hybrid_clock.cc:648] HybridClock initialized: now 1790522739012898 us; error 91 us; skew 500 ppm
I20260927 15:25:39.013502 24937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:39 dist-test-slave-14jq krb5kdc[24924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522739, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:39.023428 24937 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:39.025425 24937 webserver.cc:460] Webserver started at http://127.2.246.126:36045/ using document root <none> and password file <none>
I20260927 15:25:39.026290 24937 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:39.026407 24937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:39.026717 24937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:39.030046 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "27f67ac41d724f14bc05eb9dbee9ede6"
format_stamp: "Formatted at 2026-09-27 15:25:39 on dist-test-slave-14jq"
server_key: "cd08f8f8ea0764aba1fea664159b5e99"
server_key_iv: "4b237d17aa1e735df9fc2b61eb318ee4"
server_key_version: "encryptionkey@0"
I20260927 15:25:39.030946 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "27f67ac41d724f14bc05eb9dbee9ede6"
format_stamp: "Formatted at 2026-09-27 15:25:39 on dist-test-slave-14jq"
server_key: "cd08f8f8ea0764aba1fea664159b5e99"
server_key_iv: "4b237d17aa1e735df9fc2b61eb318ee4"
server_key_version: "encryptionkey@0"
I20260927 15:25:39.037364 24937 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 15:25:39.042117 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:39.044261 24937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:25:39.044548 24937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "27f67ac41d724f14bc05eb9dbee9ede6"
format_stamp: "Formatted at 2026-09-27 15:25:39 on dist-test-slave-14jq"
server_key: "cd08f8f8ea0764aba1fea664159b5e99"
server_key_iv: "4b237d17aa1e735df9fc2b61eb318ee4"
server_key_version: "encryptionkey@0"
I20260927 15:25:39.044803 24937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:39.069223 24937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:39.074837 24937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:39.075227 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:39.099596 24937 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:44213
I20260927 15:25:39.099644 25003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:44213 every 8 connection(s)
I20260927 15:25:39.101179 24937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:39.102583 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 24937
I20260927 15:25:39.102810 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:39.103324 3033 external_mini_cluster.cc:1428] Setting key e722d2d2c02d4e818bd48c4e3fb174b3
I20260927 15:25:39.108662 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:39.124500 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:39.105760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37043 (local address 127.2.246.126:44213)
0927 15:25:39.106898 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:39.106919 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:39.106990 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:25:39.110288 (+ 3298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:39.110338 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:39.111143 (+ 805us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:39.111828 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:39.112576 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:39.115027 (+ 2451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:39.116989 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:39.118437 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:39.118720 (+ 283us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:39.120254 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:39.120269 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:39.120295 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:39.121270 (+ 975us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:39.121435 (+ 165us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:39.121493 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:39.121734 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:39.122202 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 15:25:39.122471 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":410,"threads_started":1}
I20260927 15:25:39.127907 25004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6: Bootstrap starting.
I20260927 15:25:39.131214 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:39.133664 25004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:39.136029 25004 log.cc:824] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:39.141387 25004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6: No bootstrap required, opened a new log
I20260927 15:25:39.147509 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:39.138753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37171 (local address 127.2.246.126:44213)
0927 15:25:39.138984 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:39.138991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:39.139344 (+ 353us) server_negotiation.cc:407] Connection header received
0927 15:25:39.139570 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:39.139577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:39.139682 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:39.139867 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:39.140475 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:39.141679 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:39.143360 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:39.144196 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:39.144722 (+ 526us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:39.146280 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:39.146290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:39.146299 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:39.146340 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:39.146427 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:39.146460 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:39.146665 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:39.146946 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:25:39.147032 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:25:39.154744 25004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } }
I20260927 15:25:39.155373 25004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:39.155512 25004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f67ac41d724f14bc05eb9dbee9ede6, State: Initialized, Role: FOLLOWER
I20260927 15:25:39.156368 25004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } }
I20260927 15:25:39.156692 25004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:39.156886 25004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:39.157107 25004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:39.160629 25004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } }
I20260927 15:25:39.161434 25004 leader_election.cc:302] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f67ac41d724f14bc05eb9dbee9ede6; no voters:
I20260927 15:25:39.162508 25004 leader_election.cc:288] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:39.162576 25014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:39.164206 25014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f67ac41d724f14bc05eb9dbee9ede6, State: Running, Role: LEADER
I20260927 15:25:39.165212 25014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } }
I20260927 15:25:39.167340 25004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:39.169108 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } } }
I20260927 15:25:39.169543 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:39.168954 25015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f67ac41d724f14bc05eb9dbee9ede6. Latest consensus state: current_term: 1 leader_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f67ac41d724f14bc05eb9dbee9ede6" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 44213 } } }
I20260927 15:25:39.169723 25015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:39.172544 25019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:39.180917 25019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:39.195045 25019 catalog_manager.cc:1347] Generated new cluster ID: ec2a6701f09b4bf4906addbe6c4acdab
I20260927 15:25:39.195210 25019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:39.224110 25019 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:39.226362 25019 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:39.238801 25019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27f67ac41d724f14bc05eb9dbee9ede6: Generated new TSK 0
I20260927 15:25:39.240095 25019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:39.252617 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24937
W20260927 15:25:39.254370 24999 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37043
W20260927 15:25:39.255359 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 24937: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:39.255560 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 24937
W20260927 15:25:39.278148 24922 connection.cc:535] client connection to 127.2.246.126:44213 recv error: Network error: failed to read from TLS socket (remote: 127.2.246.126:44213): Connection reset by peer (error 104)
2026-09-27T15:25:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:39 dist-test-slave-14jq krb5kdc[25035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:39 dist-test-slave-14jq krb5kdc[25035](info): set up 2 sockets
Sep 27 15:25:39 dist-test-slave-14jq krb5kdc[25035](info): commencing operation
krb5kdc: starting...
W20260927 15:25:40.430759 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:40 dist-test-slave-14jq krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:40.734120 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37625
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:41.074417 25048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:41.074736 25048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:41.074914 25048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:41.082307 25048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:41.082453 25048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:41.082536 25048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:41.082672 25048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:41.082778 25048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:41.092962 25048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:37625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:41.094412 25048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:41.095752 25048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:41.108876 25053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:41.109138 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:41.108954 25056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:41.110265 25048 server_base.cc:1034] running on GCE node
I20260927 15:25:41.111363 25048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:41.113302 25048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:41.114545 25048 hybrid_clock.cc:648] HybridClock initialized: now 1790522741114476 us; error 59 us; skew 500 ppm
I20260927 15:25:41.114957 25048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:41 dist-test-slave-14jq krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:41.118304 25048 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:41.120083 25048 webserver.cc:460] Webserver started at http://127.2.246.126:40095/ using document root <none> and password file <none>
I20260927 15:25:41.120744 25048 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:41.120860 25048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:41.121104 25048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:41.124261 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "b119a6cbec934a788deb33cb35eb1167"
format_stamp: "Formatted at 2026-09-27 15:25:41 on dist-test-slave-14jq"
server_key: "cc4d3f5f1f9458d1cf16c68e482ae086"
server_key_iv: "6a4d29dd3c3a44107921f75e1f1e8521"
server_key_version: "encryptionkey@0"
I20260927 15:25:41.125047 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "b119a6cbec934a788deb33cb35eb1167"
format_stamp: "Formatted at 2026-09-27 15:25:41 on dist-test-slave-14jq"
server_key: "cc4d3f5f1f9458d1cf16c68e482ae086"
server_key_iv: "6a4d29dd3c3a44107921f75e1f1e8521"
server_key_version: "encryptionkey@0"
I20260927 15:25:41.131412 25048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:25:41.136374 25063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:41.138396 25048 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:25:41.138666 25048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "b119a6cbec934a788deb33cb35eb1167"
format_stamp: "Formatted at 2026-09-27 15:25:41 on dist-test-slave-14jq"
server_key: "cc4d3f5f1f9458d1cf16c68e482ae086"
server_key_iv: "6a4d29dd3c3a44107921f75e1f1e8521"
server_key_version: "encryptionkey@0"
I20260927 15:25:41.138943 25048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:41.146616 25048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:41.152350 25048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:41.152709 25048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:41.174755 25048 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:37625
I20260927 15:25:41.174868 25114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:37625 every 8 connection(s)
I20260927 15:25:41.176347 25048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:41.181396 25115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:41.186683 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25048
I20260927 15:25:41.186934 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:41.187517 3033 external_mini_cluster.cc:1428] Setting key e667157535be72fbe53ceca46200caac
I20260927 15:25:41.196171 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167: Bootstrap starting.
I20260927 15:25:41.198637 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:41.190223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49735 (local address 127.2.246.126:37625)
0927 15:25:41.191328 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:41.191345 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:41.191401 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:25:41.194356 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:41.194385 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:41.194874 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:41.195346 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:41.196707 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:41.196713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:41.196727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:41.196749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:41.196820 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:41.196843 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:41.197138 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:41.197280 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:25:41.197485 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":501,"threads_started":1}
I20260927 15:25:41.200822 25115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:41.202615 25115 log.cc:824] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:41.203063 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:41.206925 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167: No bootstrap required, opened a new log
I20260927 15:25:41.211514 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:41.208299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44291 (local address 127.2.246.126:37625)
0927 15:25:41.208557 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:41.208565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:41.208832 (+ 267us) server_negotiation.cc:407] Connection header received
0927 15:25:41.209026 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:41.209032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:41.209118 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:41.209294 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:41.210513 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:41.210521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:41.210532 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:41.210562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:41.210627 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:41.210758 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:41.210966 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:41.211040 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:25:41.211125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:25:41.218076 25115 raft_consensus.cc:348] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } }
I20260927 15:25:41.218493 25115 raft_consensus.cc:374] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:41.218616 25115 raft_consensus.cc:729] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b119a6cbec934a788deb33cb35eb1167, State: Initialized, Role: FOLLOWER
I20260927 15:25:41.219281 25115 consensus_queue.cc:260] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } }
I20260927 15:25:41.219508 25115 raft_consensus.cc:388] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:41.219698 25115 raft_consensus.cc:482] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:41.219879 25115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:41.222507 25115 raft_consensus.cc:504] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } }
I20260927 15:25:41.223206 25115 leader_election.cc:302] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b119a6cbec934a788deb33cb35eb1167; no voters:
I20260927 15:25:41.224001 25115 leader_election.cc:288] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:41.224303 25125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:41.225380 25125 raft_consensus.cc:686] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [term 1 LEADER]: Becoming Leader. State: Replica: b119a6cbec934a788deb33cb35eb1167, State: Running, Role: LEADER
I20260927 15:25:41.226176 25125 consensus_queue.cc:237] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } }
I20260927 15:25:41.226428 25115 sys_catalog.cc:564] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:41.230453 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b119a6cbec934a788deb33cb35eb1167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } } }
I20260927 15:25:41.230844 25127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b119a6cbec934a788deb33cb35eb1167. Latest consensus state: current_term: 1 leader_uuid: "b119a6cbec934a788deb33cb35eb1167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b119a6cbec934a788deb33cb35eb1167" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 37625 } } }
I20260927 15:25:41.231144 25127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:41.231144 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:41.237457 25134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:41.243731 25134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:41.255584 25134 catalog_manager.cc:1347] Generated new cluster ID: 0548137ad4e74d01b6c68d02fb6aed2e
I20260927 15:25:41.255802 25134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:41.285859 25134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:41.286525 25134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:41.297151 25134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b119a6cbec934a788deb33cb35eb1167: Generated new TSK 0
I20260927 15:25:41.298471 25134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:41.313287 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25048
W20260927 15:25:41.315140 25110 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49735
W20260927 15:25:41.316198 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25048: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:41.316390 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25048
2026-09-27T15:25:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2055 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:41 dist-test-slave-14jq krb5kdc[25146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:41 dist-test-slave-14jq krb5kdc[25146](info): set up 2 sockets
Sep 27 15:25:41 dist-test-slave-14jq krb5kdc[25146](info): commencing operation
krb5kdc: starting...
W20260927 15:25:42.462455 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:42 dist-test-slave-14jq krb5kdc[25146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:42.779244 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:40411
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:43.214534 25159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:43.214890 25159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:43.215049 25159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:43.223418 25159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:43.223577 25159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:43.223670 25159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:43.223824 25159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:43.223893 25159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:43.236210 25159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:40411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:43.237711 25159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:43.239180 25159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:43.254210 25164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:43.254262 25165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:43.254086 25167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:43.256006 25159 server_base.cc:1034] running on GCE node
I20260927 15:25:43.257045 25159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:43.258908 25159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:43.260247 25159 hybrid_clock.cc:648] HybridClock initialized: now 1790522743260192 us; error 48 us; skew 500 ppm
I20260927 15:25:43.260738 25159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:43 dist-test-slave-14jq krb5kdc[25146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:43.264811 25159 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:43.267002 25159 webserver.cc:460] Webserver started at http://127.2.246.126:33325/ using document root <none> and password file <none>
I20260927 15:25:43.267882 25159 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:43.268061 25159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:43.268426 25159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:43.272054 25159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "12eb9fd27310457aad6f1c862330f46f"
format_stamp: "Formatted at 2026-09-27 15:25:43 on dist-test-slave-14jq"
server_key: "8d3cf00496e84bf2a51eea32fe2bd22e"
server_key_iv: "bfaa9d5dc8a3392842480c2549799fdc"
server_key_version: "encryptionkey@0"
I20260927 15:25:43.272981 25159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "12eb9fd27310457aad6f1c862330f46f"
format_stamp: "Formatted at 2026-09-27 15:25:43 on dist-test-slave-14jq"
server_key: "8d3cf00496e84bf2a51eea32fe2bd22e"
server_key_iv: "bfaa9d5dc8a3392842480c2549799fdc"
server_key_version: "encryptionkey@0"
I20260927 15:25:43.280368 25159 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 15:25:43.285463 25174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:43.287487 25159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:25:43.287763 25159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "12eb9fd27310457aad6f1c862330f46f"
format_stamp: "Formatted at 2026-09-27 15:25:43 on dist-test-slave-14jq"
server_key: "8d3cf00496e84bf2a51eea32fe2bd22e"
server_key_iv: "bfaa9d5dc8a3392842480c2549799fdc"
server_key_version: "encryptionkey@0"
I20260927 15:25:43.288029 25159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:43.317917 25159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:43.322587 25159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:43.322919 25159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:43.344949 25159 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:40411
I20260927 15:25:43.345018 25225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:40411 every 8 connection(s)
I20260927 15:25:43.346602 25159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:43.352023 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:43.356611 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25159
I20260927 15:25:43.356895 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:43.357571 3033 external_mini_cluster.cc:1428] Setting key a716da2ebcc261d88f34c018d401f804
I20260927 15:25:43.368535 25226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f: Bootstrap starting.
I20260927 15:25:43.372851 25226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f: Neither blocks nor log segments found. Creating new log.
Sep 27 15:25:43 dist-test-slave-14jq krb5kdc[25146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:43.374509 25226 log.cc:824] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:43.380067 25226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f: No bootstrap required, opened a new log
I20260927 15:25:43.386904 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:43.360596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58677 (local address 127.2.246.126:40411)
0927 15:25:43.361707 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:43.361723 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:43.361774 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:25:43.363934 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:43.363965 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:43.364503 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:43.365005 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:43.366670 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:43.368065 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:43.369445 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:43.370253 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:43.370519 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:43.375405 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:43.375426 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:43.375453 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:43.375504 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:43.380492 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:43.381526 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:43.381539 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:43.381552 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:43.381896 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:43.382375 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:43.382381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:43.382384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:43.383400 (+ 1016us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:43.383815 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:43.384320 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 15:25:43.384989 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":465,"threads_started":1}
I20260927 15:25:43.391866 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:43.392465 25226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } }
I20260927 15:25:43.393075 25226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:43.393203 25226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eb9fd27310457aad6f1c862330f46f, State: Initialized, Role: FOLLOWER
I20260927 15:25:43.394043 25226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } }
I20260927 15:25:43.394378 25226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:43.394536 25226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:43.394733 25226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:43.398716 25226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } }
I20260927 15:25:43.399559 25226 leader_election.cc:302] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12eb9fd27310457aad6f1c862330f46f; no voters:
I20260927 15:25:43.400393 25226 leader_election.cc:288] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:43.400560 25236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:43.402979 25226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:43.403641 25236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [term 1 LEADER]: Becoming Leader. State: Replica: 12eb9fd27310457aad6f1c862330f46f, State: Running, Role: LEADER
I20260927 15:25:43.404517 25236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } }
I20260927 15:25:43.409528 25237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12eb9fd27310457aad6f1c862330f46f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } } }
I20260927 15:25:43.410130 25237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:43.410547 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb9fd27310457aad6f1c862330f46f. Latest consensus state: current_term: 1 leader_uuid: "12eb9fd27310457aad6f1c862330f46f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb9fd27310457aad6f1c862330f46f" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 40411 } } }
I20260927 15:25:43.410872 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:43.418558 25248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:43.423628 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:43.398272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59025 (local address 127.2.246.126:40411)
0927 15:25:43.398588 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:43.398600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:43.399181 (+ 581us) server_negotiation.cc:407] Connection header received
0927 15:25:43.399419 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:43.399423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:43.399502 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:43.399617 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:43.401914 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:43.403434 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:43.405412 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:43.406330 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:43.406586 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:43.410241 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:43.410248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:43.410254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:43.410328 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:43.419832 (+ 9504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:43.420762 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:43.420770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:43.420776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:43.421006 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:43.421565 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:43.421571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:43.421574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:43.422153 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:43.422442 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:43.422885 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 15:25:43.423193 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:25:43.425935 25248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:43.437183 25248 catalog_manager.cc:1347] Generated new cluster ID: 3afafeb686d44f5982d0a991c6b86fe8
I20260927 15:25:43.437422 25248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:43.457137 25248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:43.459300 25248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:43.470167 25248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12eb9fd27310457aad6f1c862330f46f: Generated new TSK 0
I20260927 15:25:43.471321 25248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:43.479143 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25159
I20260927 15:25:43.481061 25221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:43.586833 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25159
2026-09-27T15:25:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:43 dist-test-slave-14jq krb5kdc[25258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:43 dist-test-slave-14jq krb5kdc[25258](info): set up 2 sockets
Sep 27 15:25:43 dist-test-slave-14jq krb5kdc[25258](info): commencing operation
krb5kdc: starting...
W20260927 15:25:44.782758 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:44 dist-test-slave-14jq krb5kdc[25258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:45.096777 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46817
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:45.470729 25271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:45.471076 25271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:45.471226 25271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:45.478899 25271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:45.479043 25271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:45.479090 25271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:45.479235 25271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:45.479355 25271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:45.489655 25271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:45.491163 25271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:45.492540 25271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:45.506464 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:45.506399 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:45.506399 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:45.507493 25271 server_base.cc:1034] running on GCE node
I20260927 15:25:45.508477 25271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:45.510174 25271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:45.511422 25271 hybrid_clock.cc:648] HybridClock initialized: now 1790522745511365 us; error 56 us; skew 500 ppm
I20260927 15:25:45.511848 25271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:45 dist-test-slave-14jq krb5kdc[25258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:45.520634 25271 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:45.522645 25271 webserver.cc:460] Webserver started at http://127.2.246.126:41157/ using document root <none> and password file <none>
I20260927 15:25:45.523414 25271 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:45.523562 25271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:45.523941 25271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:45.527273 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "f5dca23a83da41389b161641c55b0806"
format_stamp: "Formatted at 2026-09-27 15:25:45 on dist-test-slave-14jq"
server_key: "cbfa7ad82f056b81f51c90dfee26e4ee"
server_key_iv: "9cb9d0f08b4f75c5cf6eefeac0d13c99"
server_key_version: "encryptionkey@0"
I20260927 15:25:45.528080 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "f5dca23a83da41389b161641c55b0806"
format_stamp: "Formatted at 2026-09-27 15:25:45 on dist-test-slave-14jq"
server_key: "cbfa7ad82f056b81f51c90dfee26e4ee"
server_key_iv: "9cb9d0f08b4f75c5cf6eefeac0d13c99"
server_key_version: "encryptionkey@0"
I20260927 15:25:45.534637 25271 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:25:45.539444 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:45.541242 25271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:25:45.541448 25271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "f5dca23a83da41389b161641c55b0806"
format_stamp: "Formatted at 2026-09-27 15:25:45 on dist-test-slave-14jq"
server_key: "cbfa7ad82f056b81f51c90dfee26e4ee"
server_key_iv: "9cb9d0f08b4f75c5cf6eefeac0d13c99"
server_key_version: "encryptionkey@0"
I20260927 15:25:45.541707 25271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:45.561821 25271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:45.566956 25271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:45.567281 25271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:45.589529 25271 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:46817
I20260927 15:25:45.589666 25337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:46817 every 8 connection(s)
I20260927 15:25:45.591069 25271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:45.596566 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:45.601595 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25271
I20260927 15:25:45.601871 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:45.602520 3033 external_mini_cluster.cc:1428] Setting key e1d050f2052f41abdf36baf5c40ccec4
I20260927 15:25:45.612982 25338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806: Bootstrap starting.
I20260927 15:25:45.617492 25338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:45.618273 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:45.605173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40693 (local address 127.2.246.126:46817)
0927 15:25:45.606414 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:45.606431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:45.606483 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:25:45.608555 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:45.608583 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:45.609106 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:45.609585 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:45.610265 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:45.611655 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:45.613012 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:45.613822 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:45.614089 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:45.615488 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:45.615496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:45.615512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:45.616187 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:45.616278 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:45.616313 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:45.616465 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:45.616699 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:25:45.617008 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":583,"threads_started":1}
I20260927 15:25:45.619251 25338 log.cc:824] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:45.623644 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:45.623714 25338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806: No bootstrap required, opened a new log
I20260927 15:25:45.635280 25338 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } }
I20260927 15:25:45.636057 25338 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:45.636194 25338 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5dca23a83da41389b161641c55b0806, State: Initialized, Role: FOLLOWER
I20260927 15:25:45.637187 25338 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } }
I20260927 15:25:45.637523 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:45.629157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44083 (local address 127.2.246.126:46817)
0927 15:25:45.629436 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:45.629443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:45.629722 (+ 279us) server_negotiation.cc:407] Connection header received
0927 15:25:45.629986 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:45.629992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:45.630109 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:45.630299 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:45.630978 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:45.632228 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:45.633828 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:45.634569 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:45.634733 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:45.636472 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:45.636478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:45.636483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:45.636514 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:45.636583 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:45.636610 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:45.636725 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:45.636965 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:25:45.637099 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:25:45.637596 25338 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:45.637787 25338 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:45.638044 25338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:45.641009 25338 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } }
I20260927 15:25:45.641726 25338 leader_election.cc:302] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5dca23a83da41389b161641c55b0806; no voters:
I20260927 15:25:45.642562 25338 leader_election.cc:288] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:45.642884 25348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:45.643893 25348 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [term 1 LEADER]: Becoming Leader. State: Replica: f5dca23a83da41389b161641c55b0806, State: Running, Role: LEADER
I20260927 15:25:45.644629 25348 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } }
I20260927 15:25:45.646090 25338 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:45.648741 25350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5dca23a83da41389b161641c55b0806. Latest consensus state: current_term: 1 leader_uuid: "f5dca23a83da41389b161641c55b0806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } } }
I20260927 15:25:45.648741 25349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5dca23a83da41389b161641c55b0806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5dca23a83da41389b161641c55b0806" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 46817 } } }
I20260927 15:25:45.649278 25349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:45.649956 25350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:45.650288 25353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:45.658005 25353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:45.671207 25353 catalog_manager.cc:1347] Generated new cluster ID: 086b2d421857449ebc1b124cd3308fdb
I20260927 15:25:45.671416 25353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:45.682597 25353 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:45.684134 25353 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:45.697163 25353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5dca23a83da41389b161641c55b0806: Generated new TSK 0
I20260927 15:25:45.698360 25353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:45.717955 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25271
W20260927 15:25:45.719471 25333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40693
W20260927 15:25:45.720382 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:45.720542 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25271
2026-09-27T15:25:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:45 dist-test-slave-14jq krb5kdc[25369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:45 dist-test-slave-14jq krb5kdc[25369](info): set up 2 sockets
Sep 27 15:25:45 dist-test-slave-14jq krb5kdc[25369](info): commencing operation
krb5kdc: starting...
W20260927 15:25:46.886633 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:47 dist-test-slave-14jq krb5kdc[25369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:47.224220 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35357
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:47.580161 25382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:47.580493 25382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:47.580590 25382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:47.588209 25382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:47.588354 25382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:47.588403 25382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:47.588541 25382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:47.588616 25382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:47.599022 25382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:47.600414 25382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:47.601807 25382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:47.614992 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:47.615103 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:47.615020 25390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:47.620208 25382 server_base.cc:1034] running on GCE node
I20260927 15:25:47.621347 25382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:47.623426 25382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:47.624729 25382 hybrid_clock.cc:648] HybridClock initialized: now 1790522747624664 us; error 73 us; skew 500 ppm
I20260927 15:25:47.625253 25382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:47 dist-test-slave-14jq krb5kdc[25369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:47.629575 25382 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:47.632143 25382 webserver.cc:460] Webserver started at http://127.2.246.126:44259/ using document root <none> and password file <none>
I20260927 15:25:47.633093 25382 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:47.633275 25382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:47.633635 25382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:47.637869 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "4db516e2647e4ee19f993968d48d12d7"
format_stamp: "Formatted at 2026-09-27 15:25:47 on dist-test-slave-14jq"
server_key: "115692f4fe9749d3f7b3a00f0a479f1f"
server_key_iv: "ae954bbfbf48ab041f7435cc0c839348"
server_key_version: "encryptionkey@0"
I20260927 15:25:47.638971 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "4db516e2647e4ee19f993968d48d12d7"
format_stamp: "Formatted at 2026-09-27 15:25:47 on dist-test-slave-14jq"
server_key: "115692f4fe9749d3f7b3a00f0a479f1f"
server_key_iv: "ae954bbfbf48ab041f7435cc0c839348"
server_key_version: "encryptionkey@0"
I20260927 15:25:47.645942 25382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:25:47.650918 25397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:47.652729 25382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:25:47.652983 25382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "4db516e2647e4ee19f993968d48d12d7"
format_stamp: "Formatted at 2026-09-27 15:25:47 on dist-test-slave-14jq"
server_key: "115692f4fe9749d3f7b3a00f0a479f1f"
server_key_iv: "ae954bbfbf48ab041f7435cc0c839348"
server_key_version: "encryptionkey@0"
I20260927 15:25:47.653231 25382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:47.659960 25382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:47.664669 25382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:47.664994 25382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:47.688546 25382 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:35357
I20260927 15:25:47.688609 25448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:35357 every 8 connection(s)
I20260927 15:25:47.690322 25382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:47.695580 25449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:47.699524 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25382
I20260927 15:25:47.699746 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:47.700313 3033 external_mini_cluster.cc:1428] Setting key 3b7cb8ded4bd63f9dd998a25206db535
I20260927 15:25:47.710422 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:47.702623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52443 (local address 127.2.246.126:35357)
0927 15:25:47.703746 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:47.703779 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:47.703888 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:25:47.706167 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:47.706196 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:47.706669 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:47.707155 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:47.708429 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:47.708436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:47.708449 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:47.708465 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:47.708530 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:47.708553 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:47.708849 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:47.709001 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:25:47.709207 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":482,"threads_started":1}
I20260927 15:25:47.711285 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7: Bootstrap starting.
I20260927 15:25:47.715754 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:47.715862 25449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:47.717185 25449 log.cc:824] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:47.722035 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7: No bootstrap required, opened a new log
I20260927 15:25:47.724571 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:47.721004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39291 (local address 127.2.246.126:35357)
0927 15:25:47.721176 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:47.721181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:47.721730 (+ 549us) server_negotiation.cc:407] Connection header received
0927 15:25:47.722019 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:47.722023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:47.722076 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:47.722203 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:47.723638 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:47.723643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:47.723646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:25:47.723663 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:47.723709 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:25:47.723724 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:47.723869 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:47.724070 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:25:47.724137 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:25:47.732900 25449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } }
I20260927 15:25:47.733467 25449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:25:47.733582 25449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db516e2647e4ee19f993968d48d12d7, State: Initialized, Role: FOLLOWER
I20260927 15:25:47.734512 25449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } }
I20260927 15:25:47.734858 25449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:25:47.735046 25449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:25:47.735249 25449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:47.739020 25449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } }
I20260927 15:25:47.739856 25449 leader_election.cc:302] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4db516e2647e4ee19f993968d48d12d7; no voters:
I20260927 15:25:47.740643 25449 leader_election.cc:288] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:25:47.741050 25459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:47.742007 25459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4db516e2647e4ee19f993968d48d12d7, State: Running, Role: LEADER
I20260927 15:25:47.742707 25459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } }
I20260927 15:25:47.743515 25449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:47.746532 25461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db516e2647e4ee19f993968d48d12d7. Latest consensus state: current_term: 1 leader_uuid: "4db516e2647e4ee19f993968d48d12d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } } }
I20260927 15:25:47.746608 25460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4db516e2647e4ee19f993968d48d12d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db516e2647e4ee19f993968d48d12d7" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 35357 } } }
I20260927 15:25:47.747282 25461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:47.747313 25460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:47.749941 25465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:47.756680 25465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:47.767884 25465 catalog_manager.cc:1347] Generated new cluster ID: eda43a72d95145c7a48b94bb30a4a7bc
I20260927 15:25:47.768042 25465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:47.789943 25465 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:47.790434 25465 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:47.798699 25465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4db516e2647e4ee19f993968d48d12d7: Generated new TSK 0
I20260927 15:25:47.799852 25465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:25:47.816226 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25382
W20260927 15:25:47.817440 25444 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52443
W20260927 15:25:47.818298 3033 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:25:47.818449 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25382
2026-09-27T15:25:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2098 ms)
[----------] 18 tests from AuthTokenIssuingTest (39105 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:47 dist-test-slave-14jq krb5kdc[25480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:47 dist-test-slave-14jq krb5kdc[25480](info): set up 2 sockets
Sep 27 15:25:47 dist-test-slave-14jq krb5kdc[25480](info): commencing operation
krb5kdc: starting...
W20260927 15:25:48.971043 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:49 dist-test-slave-14jq krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:49.290529 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:38249
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:49.638463 25493 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:49.638832 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:49.638963 25493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:49.639081 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:49.646741 25493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:49.646857 25493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:49.646943 25493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:49.647032 25493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:49.647118 25493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:49.657191 25493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:49.658633 25493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:49.659996 25493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:49.673033 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:49.673130 25501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:49.673099 25499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:49.675130 25493 server_base.cc:1034] running on GCE node
I20260927 15:25:49.676112 25493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:49.677765 25493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:49.679104 25493 hybrid_clock.cc:648] HybridClock initialized: now 1790522749679029 us; error 56 us; skew 500 ppm
I20260927 15:25:49.679493 25493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:49 dist-test-slave-14jq krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:49.683868 25493 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:49.685748 25493 webserver.cc:460] Webserver started at http://127.2.246.126:41701/ using document root <none> and password file <none>
I20260927 15:25:49.686558 25493 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:49.686653 25493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:49.686985 25493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:49.690214 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "65e9bad6ef8b4c2b97e3643164944c12"
format_stamp: "Formatted at 2026-09-27 15:25:49 on dist-test-slave-14jq"
server_key: "aafa38632ed63618832057d513a6286d"
server_key_iv: "72202b007496aac0377065ae91015cb8"
server_key_version: "encryptionkey@0"
I20260927 15:25:49.691103 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "65e9bad6ef8b4c2b97e3643164944c12"
format_stamp: "Formatted at 2026-09-27 15:25:49 on dist-test-slave-14jq"
server_key: "aafa38632ed63618832057d513a6286d"
server_key_iv: "72202b007496aac0377065ae91015cb8"
server_key_version: "encryptionkey@0"
I20260927 15:25:49.696985 25493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:25:49.701020 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:49.702831 25493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:25:49.703048 25493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "65e9bad6ef8b4c2b97e3643164944c12"
format_stamp: "Formatted at 2026-09-27 15:25:49 on dist-test-slave-14jq"
server_key: "aafa38632ed63618832057d513a6286d"
server_key_iv: "72202b007496aac0377065ae91015cb8"
server_key_version: "encryptionkey@0"
I20260927 15:25:49.703270 25493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:49.720957 25493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:49.725644 25493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:49.725986 25493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:49.748209 25493 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:38249
I20260927 15:25:49.748311 25559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:38249 every 8 connection(s)
I20260927 15:25:49.749660 25493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:49.753027 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25493
I20260927 15:25:49.753315 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:49.754359 3033 external_mini_cluster.cc:1428] Setting key 80d0124904fc1c32a90a7dff398c0247
I20260927 15:25:49.755779 25560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:49.769338 25560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.125@KRBTEST.COM" created.
Sep 27 15:25:49 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:49.804055 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.774053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38249 (local address 127.0.0.1:40368)
0927 15:25:49.775402 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.775495 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:49.778066 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:49.779343 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:49.779350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:49.780174 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:49.780652 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:49.780675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:49.783287 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:49.783292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:49.784849 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:49.784866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:49.786600 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:49.786607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:49.786864 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:49.787732 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:49.787770 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:49.793239 (+ 5469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:49.798788 (+ 5549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:49.798807 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:49.798822 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:49.799395 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:49.800076 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:49.800080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:49.800083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:49.800257 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:49.801666 (+ 1409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:49.801673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:49.802011 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 15:25:49.802261 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 15:25:49.802741 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":60,"thread_start_us":484,"threads_started":1}
I20260927 15:25:49.804114 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.774052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.2.246.126:38249)
0927 15:25:49.776128 (+ 2076us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:49.776140 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:49.776201 (+ 61us) server_negotiation.cc:407] Connection header received
0927 15:25:49.778589 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:49.778601 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:49.778980 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:49.779245 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:49.780953 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:49.783045 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:49.785058 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:49.786265 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:49.786524 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:49.793666 (+ 7142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:49.793685 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:49.793717 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:49.793775 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:49.798557 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:49.799617 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:49.799631 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:49.799638 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:49.799903 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:49.800412 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:49.800421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:49.800426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:49.801461 (+ 1035us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:49.801710 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:49.802479 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 15:25:49.802980 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":1811,"spinlock_wait_cycles":19840,"thread_start_us":1497,"threads_started":1}
I20260927 15:25:49.808421 25560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:49.810158 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.809530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41674)
0927 15:25:49.809779 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.809968 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 15:25:49.810552 25511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
I20260927 15:25:49.812593 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.812090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41680)
0927 15:25:49.812293 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.812405 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 15:25:49.812907 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:45775: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
I20260927 15:25:49.833802 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:45775
--webserver_interface=127.2.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:25:49.862418 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } attempt: 1
I20260927 15:25:49.864234 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.863525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41694)
0927 15:25:49.863856 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.864032 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 15:25:49.865583 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.865129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41708)
0927 15:25:49.865303 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.865410 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 15:25:49.865940 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:45775: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
I20260927 15:25:49.934305 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } attempt: 2
I20260927 15:25:49.935925 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.935250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41720)
0927 15:25:49.935606 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.935744 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 15:25:49.937325 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:49.936861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41722)
0927 15:25:49.937062 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:49.937159 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 15:25:49.937652 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:45775: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
I20260927 15:25:50.029093 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } attempt: 3
I20260927 15:25:50.030735 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.030081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41724)
0927 15:25:50.030382 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.030541 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 15:25:50.032119 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.031657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41740)
0927 15:25:50.031853 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.031948 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 15:25:50.032418 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:45775: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
I20260927 15:25:50.175735 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } attempt: 4
I20260927 15:25:50.177870 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.176868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41756)
0927 15:25:50.177421 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.177595 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
I20260927 15:25:50.179674 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.178997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41762)
0927 15:25:50.179317 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.179465 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260927 15:25:50.180050 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:45775: Network error: Client connection negotiation failed: client connection to 127.2.246.125:45775: connect: Connection refused (error 111)
W20260927 15:25:50.186590 25567 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:50.186957 25567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:50.187112 25567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:50.187255 25567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:50.195125 25567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:50.195250 25567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:50.195312 25567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:50.195401 25567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:50.195498 25567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:50.206063 25567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:50.207782 25567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:50.209183 25567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:50.223505 25572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:50.223480 25573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:50.223707 25575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:50.224119 25567 server_base.cc:1034] running on GCE node
I20260927 15:25:50.225231 25567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:50.227048 25567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:50.228448 25567 hybrid_clock.cc:648] HybridClock initialized: now 1790522750228333 us; error 58 us; skew 500 ppm
I20260927 15:25:50.229243 25567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:50 dist-test-slave-14jq krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:50.233619 25567 init.cc:377] Logged in from keytab as kudu/127.2.246.125@KRBTEST.COM (short username kudu)
I20260927 15:25:50.235828 25567 webserver.cc:460] Webserver started at http://127.2.246.125:37951/ using document root <none> and password file <none>
I20260927 15:25:50.236681 25567 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:50.236824 25567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:50.237187 25567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:50.240511 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data/instance:
uuid: "d407e30a0685484cb4ff76cce9cedc43"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "5d4c4d5e7fc7ed5fdc7f8c9c153e53c2"
server_key_iv: "6ff586feb8ca0eea25ff6bbc2b1d2289"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.241350 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal/instance:
uuid: "d407e30a0685484cb4ff76cce9cedc43"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "5d4c4d5e7fc7ed5fdc7f8c9c153e53c2"
server_key_iv: "6ff586feb8ca0eea25ff6bbc2b1d2289"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.247321 25567 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 15:25:50.251649 25582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:50.253309 25567 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:25:50.253526 25567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "d407e30a0685484cb4ff76cce9cedc43"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "5d4c4d5e7fc7ed5fdc7f8c9c153e53c2"
server_key_iv: "6ff586feb8ca0eea25ff6bbc2b1d2289"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.253769 25567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:50.290938 25567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:50.295486 25567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:50.295787 25567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:50.317763 25567 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.125:45775
I20260927 15:25:50.317827 25633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.125:45775 every 8 connection(s)
I20260927 15:25:50.319329 25567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
I20260927 15:25:50.324501 25634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:50.329710 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25567
I20260927 15:25:50.329993 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-1/wal/instance
I20260927 15:25:50.330633 3033 external_mini_cluster.cc:1428] Setting key 7766677455edc775f655a6b63f1479e8
I20260927 15:25:50.335237 25634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:50 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.124@KRBTEST.COM" created.
I20260927 15:25:50.369145 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.340282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.2.246.126:38249)
0927 15:25:50.340976 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:50.340984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:50.342421 (+ 1437us) server_negotiation.cc:407] Connection header received
0927 15:25:50.346473 (+ 4052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:50.346482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:50.346603 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:50.346803 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:50.349198 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.350444 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.352622 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.353383 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.353578 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.358640 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:50.358652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:50.358661 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:50.358741 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:50.363315 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.364683 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.364690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.364695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.364871 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.365681 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.365686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.365689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.366341 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:50.366569 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:50.368205 (+ 1636us) server_negotiation.cc:299] Negotiation successful
0927 15:25:50.368617 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260927 15:25:50.370633 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.340871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38249 (local address 127.0.0.1:40372)
0927 15:25:50.342115 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.342199 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:50.345999 (+ 3800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:50.347111 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:50.347122 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:50.347929 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:50.348858 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.348884 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.350704 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.350710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.352378 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.352398 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.353705 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.353711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.354099 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.355112 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:50.355154 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:50.358231 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:50.363616 (+ 5385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.363629 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.363651 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.364467 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.365084 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.365089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.365093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.365417 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.366536 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:50.366549 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:50.367438 (+ 889us) client_negotiation.cc:769] Sending connection context
0927 15:25:50.367807 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 15:25:50.368853 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":419,"threads_started":1}
I20260927 15:25:50.373369 25634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:50 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:25:50.393450 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.374612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41772)
0927 15:25:50.374835 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.374862 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:50.375068 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:50.376062 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:50.376067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:50.376654 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:50.376820 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.376830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.378431 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.378439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.379359 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.379371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.380334 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.380340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.380550 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.381389 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:50.381409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:50.384403 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:50.389348 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.389358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.389363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.390162 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.390794 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.390804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.390811 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.391086 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.392035 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:50.392043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:50.392268 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 15:25:50.392486 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 15:25:50.392858 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 15:25:50.393594 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.374741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.2.246.125:45775)
0927 15:25:50.375502 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:50.375510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:50.375542 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:25:50.375602 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:50.375610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:50.375879 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:50.376082 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:50.377018 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.378292 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.379512 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.380167 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.380315 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.384779 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:50.384798 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:50.384815 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:50.384853 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:50.389040 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.390404 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.390415 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.390418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.390626 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.391267 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.391272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.391275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.391837 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:50.392083 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:50.392722 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 15:25:50.393120 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":377,"threads_started":1}
I20260927 15:25:50.397085 25634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:50.398770 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.398087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57320)
0927 15:25:50.398378 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.398582 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 15:25:50.399155 25585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.400990 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.400599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57326)
0927 15:25:50.400768 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.400872 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:25:50.401211 25634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
I20260927 15:25:50.406318 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:37065
--webserver_interface=127.2.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:25:50.450536 25634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 1
I20260927 15:25:50.451730 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.451277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57330)
0927 15:25:50.451472 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.451581 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:25:50.452677 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.452340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57340)
0927 15:25:50.452504 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.452574 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:25:50.452878 25634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.479386 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } attempt: 5
Sep 27 15:25:50 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:25:50.494390 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.480431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41786)
0927 15:25:50.480631 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.480653 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:50.480814 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:50.481206 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:50.481211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:50.481669 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:50.481984 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.481995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.483008 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.483016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.484024 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:50.484036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.484938 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.484943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:50.485167 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.486033 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:50.486049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:50.488561 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:50.491922 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.491928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.491932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.492304 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.492676 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:50.492682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:50.492686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:50.492821 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:50.493357 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:50.493362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:50.493520 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 15:25:50.493697 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:25:50.494022 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:25:50.494390 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.480626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.2.246.125:45775)
0927 15:25:50.480875 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:50.480882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:50.480905 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:25:50.480978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:50.480981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:50.481068 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:50.481178 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:50.482140 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.482873 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.484215 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:50.484752 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:50.484924 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:50.488771 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:50.488776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:50.488779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:50.488810 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:50.491777 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.492432 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.492438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.492442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.492565 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:50.492936 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:50.492942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:50.492945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:50.493235 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:50.493402 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:50.493791 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 15:25:50.494059 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:25:50.495909 25560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:50.497241 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.496845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57356)
0927 15:25:50.497019 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.497095 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:25:50.498368 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.498025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57364)
0927 15:25:50.498164 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.498248 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:25:50.498689 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.521123 25634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 2
I20260927 15:25:50.522742 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.521942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57368)
0927 15:25:50.522295 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.522463 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 15:25:50.524123 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.523585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57382)
0927 15:25:50.523814 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.523945 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 15:25:50.524446 25634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.550084 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 1
I20260927 15:25:50.551545 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.551034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57390)
0927 15:25:50.551247 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.551352 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:25:50.552529 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.552213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57400)
0927 15:25:50.552353 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.552429 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:25:50.552754 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.615746 25634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 3
I20260927 15:25:50.617545 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.616637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57404)
0927 15:25:50.617050 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.617233 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 15:25:50.619047 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.618404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57410)
0927 15:25:50.618755 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.618890 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260927 15:25:50.619333 25634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.620988 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 2
I20260927 15:25:50.622150 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.621583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57416)
0927 15:25:50.621836 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.621982 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 15:25:50.623150 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.622773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57424)
0927 15:25:50.622971 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.623049 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:25:50.623430 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.729720 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 3
I20260927 15:25:50.731348 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.730512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57438)
0927 15:25:50.730873 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.731056 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 15:25:50.732815 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.732322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57440)
0927 15:25:50.732564 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.732670 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 15:25:50.733100 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
I20260927 15:25:50.762598 25634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 4
I20260927 15:25:50.764451 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.763469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57454)
0927 15:25:50.763943 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.764144 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 15:25:50.766038 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.765356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57458)
0927 15:25:50.765628 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.765776 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 15:25:50.766415 25634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
W20260927 15:25:50.852263 25642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:50.852943 25642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:50.853086 25642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:50.853315 25642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:50.864841 25642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:50.864995 25642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:50.865062 25642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:50.865124 25642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:50.865213 25642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:50.878788 25642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:38249,127.2.246.125:45775,127.2.246.124:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:50.880546 25642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:50.882364 25642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:50.898473 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:50.898438 25647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:50.898535 25648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:50.900053 25642 server_base.cc:1034] running on GCE node
I20260927 15:25:50.901180 25642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:50.903167 25642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:50.904470 25642 hybrid_clock.cc:648] HybridClock initialized: now 1790522750904346 us; error 77 us; skew 500 ppm
I20260927 15:25:50.904965 25642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:25:50.910349 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 4
I20260927 15:25:50.911666 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.911152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57464)
0927 15:25:50.911390 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.911499 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:25:50.912765 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:50.912402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57466)
0927 15:25:50.912563 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:50.912633 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:25:50.913043 25560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37065: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37065: connect: Connection refused (error 111)
Sep 27 15:25:50 dist-test-slave-14jq krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:50.914318 25642 init.cc:377] Logged in from keytab as kudu/127.2.246.124@KRBTEST.COM (short username kudu)
I20260927 15:25:50.916424 25642 webserver.cc:460] Webserver started at http://127.2.246.124:46677/ using document root <none> and password file <none>
I20260927 15:25:50.917311 25642 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:50.917418 25642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:50.917773 25642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:50.921479 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data/instance:
uuid: "7d0dd5c114cb4543a2c1acc9d92838db"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "72ed234e791655f0b731107690328d09"
server_key_iv: "ce07c4805f0df8b9c43b2a21dfa9475f"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.922540 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal/instance:
uuid: "7d0dd5c114cb4543a2c1acc9d92838db"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "72ed234e791655f0b731107690328d09"
server_key_iv: "ce07c4805f0df8b9c43b2a21dfa9475f"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.929955 25642 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 15:25:50.934942 25657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:50.937057 25642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:25:50.937359 25642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "7d0dd5c114cb4543a2c1acc9d92838db"
format_stamp: "Formatted at 2026-09-27 15:25:50 on dist-test-slave-14jq"
server_key: "72ed234e791655f0b731107690328d09"
server_key_iv: "ce07c4805f0df8b9c43b2a21dfa9475f"
server_key_version: "encryptionkey@0"
I20260927 15:25:50.937696 25642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:50.964915 25642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:50.970925 25642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:50.971376 25642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:50.995781 25642 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.124:37065
I20260927 15:25:50.995903 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.124:37065 every 8 connection(s)
I20260927 15:25:50.997649 25642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
I20260927 15:25:50.999894 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25642
I20260927 15:25:51.000139 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522351366233-3033-0/minicluster-data/master-2/wal/instance
I20260927 15:25:51.000685 3033 external_mini_cluster.cc:1428] Setting key 58c70964533c7fda9d1b3a5cba18a723
I20260927 15:25:51.002351 25709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:51.013245 25709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:51.031829 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.003046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.2.246.124:37065)
0927 15:25:51.004107 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.004122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.004177 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:25:51.006281 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.006315 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.007012 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.007596 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.009911 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.011750 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.013809 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.014959 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.015162 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.021231 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.021249 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.021276 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.021351 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.026588 (+ 5237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.027587 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.027596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.027605 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.027837 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.028265 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.028273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.028281 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.028957 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.029315 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.029766 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.030298 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":470,"threads_started":1}
I20260927 15:25:51.036760 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.016486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:38249 (local address 127.0.0.1:40386)
0927 15:25:51.017358 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:51.017401 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:51.017581 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:51.018473 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:51.018481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:51.019173 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:51.019445 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.019459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.021048 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.021063 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.023181 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.023206 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.024316 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.024325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.024550 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.025315 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:51.025343 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:51.028858 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:51.033339 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.033351 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.033361 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.034173 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.034653 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.034658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.034661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.034878 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.035564 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:51.035573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:51.035761 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 15:25:51.035959 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:25:51.036338 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":445,"threads_started":1}
I20260927 15:25:51.037905 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.016595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.2.246.126:38249)
0927 15:25:51.018118 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.018124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.018145 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:25:51.018213 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.018217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.018289 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.018411 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.019605 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.020744 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.023421 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.024063 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.024174 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.029184 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.029192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.029195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.029243 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.033088 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.034351 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.034356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.034360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.034521 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.035037 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.035044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.035048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.035418 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.035586 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.036943 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.037291 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1197,"threads_started":1}
I20260927 15:25:51.039592 25709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:25:51.055378 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.036172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.2.246.126:38249)
0927 15:25:51.037030 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.037036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.037063 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:25:51.037120 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.037125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.037213 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.037351 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.038988 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.040043 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.041785 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.042395 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.042545 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.048066 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.048073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.048077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.048116 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.051558 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.052838 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.052844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.052848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.053002 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.053514 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.053518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.053520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.053822 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.054124 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.054667 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.054922 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":394,"threads_started":1}
I20260927 15:25:51.057986 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.040309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:45775 (local address 127.0.0.1:41792)
0927 15:25:51.040499 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:51.040520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:51.040617 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:51.041429 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:51.041434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:51.041918 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:51.042072 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.042080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.043395 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.043403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.045000 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.045018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.045979 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.045987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.046140 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.046920 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:51.046947 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:51.049489 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:51.054443 (+ 4954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.054454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.054457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.055089 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.055748 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.055754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.055757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.055973 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.056660 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:51.056666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:51.056883 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 15:25:51.057060 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:25:51.057472 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 15:25:51.058161 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.040518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.2.246.125:45775)
0927 15:25:51.041147 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.041151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.041166 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:25:51.041214 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.041217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.041272 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.041399 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.042275 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.043252 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.045221 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.045710 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.045937 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.049986 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.049993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.049997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.050044 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.054215 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.055277 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.055286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.055290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.055492 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.056123 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.056131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.056135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.056497 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.056710 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.057172 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.057472 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":381,"threads_started":1}
I20260927 15:25:51.059419 25709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:51.065678 25634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 5
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522750, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:25:51.080778 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.057373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.2.246.125:45775)
0927 15:25:51.058446 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.058452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.058471 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:25:51.058525 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.058530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.058600 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.058749 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.060390 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.061997 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.064716 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.065318 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.065496 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.072762 (+ 7266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.072774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.072781 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.072847 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.077012 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.078127 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.078133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.078137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.078344 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.078843 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.078850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.078854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.079341 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.079678 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.080017 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.080329 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":9210,"threads_started":1}
I20260927 15:25:51.081041 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.060359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57482)
0927 15:25:51.060639 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:51.060665 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:51.060822 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:51.061667 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:51.061673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:51.062282 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:51.062483 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.062493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.063578 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.063586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.064552 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.064565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.066331 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.066340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.066648 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.070103 (+ 3455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:51.070142 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:51.073755 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:51.078010 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.078018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.078024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.078601 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.079093 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.079103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.079109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.079270 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.079959 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:51.079963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:51.080116 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 15:25:51.080315 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 15:25:51.080671 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 15:25:51.081954 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.060947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.2.246.124:37065)
0927 15:25:51.061243 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.061252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.061279 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:25:51.061349 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.061357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.061463 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.061607 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.062658 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.063423 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.064858 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.065823 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.066033 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.073989 (+ 7956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.073996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.074001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.074038 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.077749 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.078763 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.078772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.078778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.078956 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.079387 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.079394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.079400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.079754 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.079907 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.081307 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.081578 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:25:51.085343 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.067092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57492)
0927 15:25:51.067374 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:51.067402 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:51.067547 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:51.068074 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:51.068082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:51.068564 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:51.068743 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.068752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.070099 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.070106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.071062 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.071074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.071906 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.071912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.072178 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.073103 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:51.073130 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:51.076949 (+ 3819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:51.081721 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.081730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.081734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.082387 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.082834 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.082839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.082843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.083018 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.084296 (+ 1278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:51.084302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:51.084515 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 15:25:51.084698 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 15:25:51.085031 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:25:51.085577 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.067053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.2.246.124:37065)
0927 15:25:51.067765 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.067769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.067790 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:25:51.067833 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.067836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.067889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.068021 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.068917 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.069930 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.071228 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.071729 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.071880 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.077325 (+ 5445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.077332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.077336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.077382 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.081550 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.082523 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.082532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.082535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.082710 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.083161 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.083166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.083169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.084151 (+ 982us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.084363 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.084850 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.085098 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":409,"threads_started":1}
I20260927 15:25:51.088337 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: Bootstrap starting.
I20260927 15:25:51.091634 25709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:51.091701 25634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: Bootstrap starting.
I20260927 15:25:51.092830 25709 log.cc:824] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:51.094937 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:51.096979 25634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:51.097069 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: No bootstrap required, opened a new log
I20260927 15:25:51.098781 25634 log.cc:824] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:51.097658 25535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "65e9bad6ef8b4c2b97e3643164944c12"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40396
I20260927 15:25:51.103622 25634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: No bootstrap required, opened a new log
I20260927 15:25:51.110287 25709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.110926 25709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0dd5c114cb4543a2c1acc9d92838db, State: Initialized, Role: FOLLOWER
I20260927 15:25:51.111613 25709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.113045 25721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.113570 25721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.114723 25709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:51.116636 25634 raft_consensus.cc:348] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.117244 25634 raft_consensus.cc:729] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d407e30a0685484cb4ff76cce9cedc43, State: Initialized, Role: FOLLOWER
I20260927 15:25:51.118072 25634 consensus_queue.cc:260] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.119282 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.119783 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.121474 25634 sys_catalog.cc:564] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:25:51.147437 25744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:25:51.147748 25744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:25:51.134258 25642 master_runner.cc:186] Error getting master registration for 127.2.246.126:38249: OK, instance_id { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" instance_seqno: 1790522749742200 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:25:51.132586 25567 master_runner.cc:186] Error getting master registration for 127.2.246.126:38249: OK, instance_id { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" instance_seqno: 1790522749742200 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:25:51.150034 25642 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:25:51.150468 25567 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:25:51.153606 25743 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:25:51.153766 25743 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:25:51.190358 25560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } attempt: 5
Sep 27 15:25:51 dist-test-slave-14jq krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522749, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:25:51.210140 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.191355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37065 (local address 127.0.0.1:57494)
0927 15:25:51.191837 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:51.191859 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:51.191990 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:51.192927 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:51.192932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:51.193463 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:51.193678 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.193689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.195219 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.195224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.196749 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:51.196765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.198004 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.198011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:51.198215 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.198994 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:51.199022 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:51.201977 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:51.206850 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.206856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.206860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.207417 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.207969 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:51.207976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:51.207982 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:51.208173 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:51.209007 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:51.209015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:51.209188 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 15:25:51.209343 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 15:25:51.209650 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324}
I20260927 15:25:51.211114 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.192220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.2.246.124:37065)
0927 15:25:51.192414 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.192420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.192438 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:25:51.192503 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.192508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.192616 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.192724 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.193907 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.195041 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.196943 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.197621 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.197741 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.202252 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.202260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.202265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.202312 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.206536 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.207571 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.207577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.207581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.207834 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.208304 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.208311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.208315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.208712 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.208939 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.210479 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.210746 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:25:51.218773 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12: Bootstrap starting.
W20260927 15:25:51.224151 25663 tablet.cc:2369] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:51.224249 25560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:51.226183 25560 log.cc:824] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:51.230930 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12: No bootstrap required, opened a new log
I20260927 15:25:51.240950 25560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.241626 25560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e9bad6ef8b4c2b97e3643164944c12, State: Initialized, Role: FOLLOWER
I20260927 15:25:51.242511 25560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.244326 25748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.245222 25748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.245913 25560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:51.246008 25535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:25:51.246272 25535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:51.250281 25535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.253005 25609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65e9bad6ef8b4c2b97e3643164944c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d407e30a0685484cb4ff76cce9cedc43"
I20260927 15:25:51.253510 25609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:51.257494 25609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e9bad6ef8b4c2b97e3643164944c12 in term 1.
I20260927 15:25:51.258476 25511 leader_election.cc:302] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e9bad6ef8b4c2b97e3643164944c12, d407e30a0685484cb4ff76cce9cedc43; no voters:
I20260927 15:25:51.260252 25684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65e9bad6ef8b4c2b97e3643164944c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d0dd5c114cb4543a2c1acc9d92838db"
I20260927 15:25:51.260807 25684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:51.264623 25684 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e9bad6ef8b4c2b97e3643164944c12 in term 1.
I20260927 15:25:51.265462 25535 leader_election.cc:288] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [CANDIDATE]: Term 1 election: Requested vote from peers d407e30a0685484cb4ff76cce9cedc43 (127.2.246.125:45775), 7d0dd5c114cb4543a2c1acc9d92838db (127.2.246.124:37065)
I20260927 15:25:51.266505 25748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:51.270906 25748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [term 1 LEADER]: Becoming Leader. State: Replica: 65e9bad6ef8b4c2b97e3643164944c12, State: Running, Role: LEADER
I20260927 15:25:51.271867 25748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } }
I20260927 15:25:51.282579 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e9bad6ef8b4c2b97e3643164944c12. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.283349 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:51.294606 25755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 15:25:51.299352 25588 tablet.cc:2369] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:51.322342 25755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:51.327144 25766 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:25:51.364710 25684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [term 1 FOLLOWER]: Refusing update from remote peer 65e9bad6ef8b4c2b97e3643164944c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:51.367726 25609 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [term 1 FOLLOWER]: Refusing update from remote peer 65e9bad6ef8b4c2b97e3643164944c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:51.369202 25751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:51.370357 25748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:51.407878 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e9bad6ef8b4c2b97e3643164944c12. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.408255 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.409482 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.346213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.2.246.126:38249)
0927 15:25:51.346522 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.346530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.350702 (+ 4172us) server_negotiation.cc:407] Connection header received
0927 15:25:51.351064 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.351070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.351182 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.351267 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.359873 (+ 8606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.361109 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.383131 (+ 22022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.384107 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.384271 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.398400 (+ 14129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.398410 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.398416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.398486 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.402894 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.405196 (+ 2302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.405204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.405209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.405440 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.407073 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.407082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.407085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.407599 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.407904 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.408718 (+ 814us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.409106 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 15:25:51.409952 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.345705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.2.246.125:45775)
0927 15:25:51.345946 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.345953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.354245 (+ 8292us) server_negotiation.cc:407] Connection header received
0927 15:25:51.355364 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.355371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.355473 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.355556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.362262 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.363492 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.370785 (+ 7293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.371486 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.371590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.390667 (+ 19077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.390676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.390682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.390778 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.395001 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.400988 (+ 5987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.400997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.401005 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.401199 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.406965 (+ 5766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.406973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.406978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.407508 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.407898 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.409232 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.409506 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:25:51.413189 25768 mvcc.cc:205] Tried to move back new op lower bound from 7333981189564891136 to 7333981189216509952. Current Snapshot: MvccSnapshot[applied={T|T < 7333981189564891136}]
I20260927 15:25:51.415129 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.415390 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.418262 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:51.350972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.2.246.124:37065)
0927 15:25:51.351219 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.351228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.359456 (+ 8228us) server_negotiation.cc:407] Connection header received
0927 15:25:51.359770 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.359776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.359877 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.359970 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.367093 (+ 7123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.368272 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.381487 (+ 13215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:51.382327 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:51.382452 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:51.401788 (+ 19336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:51.401795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:51.401800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:51.401883 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:51.405938 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.408761 (+ 2823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.408767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.408772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.408944 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:51.413266 (+ 4322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:51.413272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:51.413276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:51.413689 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:51.414060 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:51.417513 (+ 3453us) server_negotiation.cc:299] Negotiation successful
0927 15:25:51.417864 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:25:51.425254 25721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e9bad6ef8b4c2b97e3643164944c12. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.425580 25721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.428861 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.429258 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:51.429682 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.429932 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:51.430231 25772 mvcc.cc:205] Tried to move back new op lower bound from 7333981189564891136 to 7333981189216509952. Current Snapshot: MvccSnapshot[applied={T|T < 7333981189564891136}]
I20260927 15:25:51.437770 25721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65e9bad6ef8b4c2b97e3643164944c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e9bad6ef8b4c2b97e3643164944c12" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 38249 } } peers { permanent_uuid: "d407e30a0685484cb4ff76cce9cedc43" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 45775 } } peers { permanent_uuid: "7d0dd5c114cb4543a2c1acc9d92838db" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37065 } } }
I20260927 15:25:51.438095 25721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:51.441684 25755 catalog_manager.cc:1347] Generated new cluster ID: c919a431f2f24c99bf62f9ed43933719
I20260927 15:25:51.441826 25755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:51.470489 25755 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:51.472947 25755 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:51.492074 25755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e9bad6ef8b4c2b97e3643164944c12: Generated new TSK 0
I20260927 15:25:51.493753 25755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:25:51.639482 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.612087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.2.246.126:38249)
0927 15:25:51.612310 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.612317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.633244 (+ 20927us) server_negotiation.cc:407] Connection header received
0927 15:25:51.636585 (+ 3341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.636592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.636701 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.636800 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.639111 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40406: BlockingRecv error: recv got EOF from 127.0.0.1:40406 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 15:25:51.642195 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.618782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.2.246.125:45775)
0927 15:25:51.619042 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.619049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.637594 (+ 18545us) server_negotiation.cc:407] Connection header received
0927 15:25:51.640420 (+ 2826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.640427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.640520 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.640617 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.641893 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41810: BlockingRecv error: recv got EOF from 127.0.0.1:41810 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260927 15:25:51.645238 25718 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.634048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.2.246.124:37065)
0927 15:25:51.634291 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.634299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.643075 (+ 8776us) server_negotiation.cc:407] Connection header received
0927 15:25:51.643615 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.643622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.643745 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.643871 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.644932 (+ 1061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57506: BlockingRecv error: recv got EOF from 127.0.0.1:57506 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 15:25:51.688110 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.677436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.2.246.125:45775)
0927 15:25:51.677644 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.677650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.685596 (+ 7946us) server_negotiation.cc:407] Connection header received
0927 15:25:51.686290 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.686298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.686442 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.686532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.687802 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41824: BlockingRecv error: recv got EOF from 127.0.0.1:41824 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:25:51.744872 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.724425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.2.246.125:45775)
0927 15:25:51.724633 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.724640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.742690 (+ 18050us) server_negotiation.cc:407] Connection header received
0927 15:25:51.743086 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.743093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.743207 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.743298 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.744582 (+ 1284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41834: BlockingRecv error: recv got EOF from 127.0.0.1:41834 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 15:25:51.804988 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.795822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.2.246.125:45775)
0927 15:25:51.796014 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.796021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.802680 (+ 6659us) server_negotiation.cc:407] Connection header received
0927 15:25:51.803113 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.803128 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.803265 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.803359 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.804670 (+ 1311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41850: BlockingRecv error: recv got EOF from 127.0.0.1:41850 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 15:25:51.819295 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.816401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.2.246.125:45775)
0927 15:25:51.816719 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.816725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.817167 (+ 442us) server_negotiation.cc:407] Connection header received
0927 15:25:51.817422 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.817429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.817541 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.817672 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.818928 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41852: BlockingRecv error: recv got EOF from 127.0.0.1:41852 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
W20260927 15:25:51.836233 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.833157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.2.246.125:45775)
0927 15:25:51.833414 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.833423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.834172 (+ 749us) server_negotiation.cc:407] Connection header received
0927 15:25:51.834455 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.834465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.834605 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.834749 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.835965 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41854: BlockingRecv error: recv got EOF from 127.0.0.1:41854 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 15:25:51.861546 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.858055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.2.246.125:45775)
0927 15:25:51.858540 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.858572 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.859652 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 15:25:51.859916 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.859938 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.860069 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.860199 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.861311 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41866: BlockingRecv error: recv got EOF from 127.0.0.1:41866 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260927 15:25:51.900300 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.897740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.2.246.125:45775)
0927 15:25:51.898122 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.898133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.898440 (+ 307us) server_negotiation.cc:407] Connection header received
0927 15:25:51.898642 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.898650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.898777 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.898970 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.899962 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41868: BlockingRecv error: recv got EOF from 127.0.0.1:41868 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
W20260927 15:25:51.970574 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:51.967891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.2.246.125:45775)
0927 15:25:51.968115 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:51.968123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:51.968538 (+ 415us) server_negotiation.cc:407] Connection header received
0927 15:25:51.968864 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:51.968870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:51.968961 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:51.969071 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:51.970302 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41870: BlockingRecv error: recv got EOF from 127.0.0.1:41870 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 15:25:52.105400 25716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:52.103400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.2.246.125:45775)
0927 15:25:52.103604 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.103609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.104031 (+ 422us) server_negotiation.cc:407] Connection header received
0927 15:25:52.104220 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.104224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.104304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.104445 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:52.105199 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41878: BlockingRecv error: recv got EOF from 127.0.0.1:41878 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:25:52.150442 25744 catalog_manager.cc:1259] Loaded cluster ID: c919a431f2f24c99bf62f9ed43933719
I20260927 15:25:52.150656 25744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: loading cluster ID for follower catalog manager: success
I20260927 15:25:52.154102 25744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: acquiring CA information for follower catalog manager: success
I20260927 15:25:52.156008 25743 catalog_manager.cc:1259] Loaded cluster ID: c919a431f2f24c99bf62f9ed43933719
I20260927 15:25:52.156112 25743 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: loading cluster ID for follower catalog manager: success
I20260927 15:25:52.156967 25744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d0dd5c114cb4543a2c1acc9d92838db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:25:52.159026 25743 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: acquiring CA information for follower catalog manager: success
I20260927 15:25:52.160897 25743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d407e30a0685484cb4ff76cce9cedc43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:25:52.373974 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:52.366129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.2.246.125:45775)
0927 15:25:52.366488 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.366494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.366723 (+ 229us) server_negotiation.cc:407] Connection header received
0927 15:25:52.366936 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.366942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.367116 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.367249 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:52.367849 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.368877 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.370759 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.371287 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.371435 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:52.372144 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:52.372956 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:52.373125 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:52.373291 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:25:52.373399 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 15:25:52.389353 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:52.381984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.2.246.124:37065)
0927 15:25:52.383432 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.383436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.383452 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:25:52.383555 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.383559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.383714 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.383894 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:52.384486 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.385224 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.386836 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.387226 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.387337 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:52.388002 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:52.388627 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:52.388754 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:52.388988 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:25:52.389064 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":1207,"threads_started":1}
I20260927 15:25:52.415150 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:52.401381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.2.246.125:45775)
0927 15:25:52.401594 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.401603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.407885 (+ 6282us) server_negotiation.cc:407] Connection header received
0927 15:25:52.408219 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.408227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.408357 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.408477 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:52.409092 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.410359 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.412702 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.413220 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.413387 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:52.414117 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:52.414271 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:52.414384 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:52.414655 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:25:52.414766 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:25:52.415591 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:52.401374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.2.246.126:38249)
0927 15:25:52.406284 (+ 4910us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.406292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.406721 (+ 429us) server_negotiation.cc:407] Connection header received
0927 15:25:52.407953 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.407960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.408262 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.408464 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:52.409495 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.410463 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.412412 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.412958 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.413097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:52.413763 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:52.414370 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:52.414498 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:52.415122 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 15:25:52.415246 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":4659,"threads_started":1}
I20260927 15:25:52.416076 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:52.407298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.2.246.124:37065)
0927 15:25:52.407553 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:52.407558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:52.408100 (+ 542us) server_negotiation.cc:407] Connection header received
0927 15:25:52.408360 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:52.408366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:52.408471 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:52.408557 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:52.409737 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.410910 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.413963 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:52.414628 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:52.414745 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:52.415352 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:52.415486 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:52.415562 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:52.415748 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:25:52.415832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 15:25:52.421980 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25493
I20260927 15:25:52.423409 25555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:52.563838 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25493
W20260927 15:25:52.588132 25583 connection.cc:535] server connection from 127.0.0.1:41786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:25:52.588701 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25567
I20260927 15:25:52.590298 25629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:52.704800 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25567
I20260927 15:25:52.726081 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25642
I20260927 15:25:52.727571 25704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:52.854624 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25642
2026-09-27T15:25:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5039 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:25:52 dist-test-slave-14jq krb5kdc[25857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:25:52 dist-test-slave-14jq krb5kdc[25857](info): set up 2 sockets
Sep 27 15:25:52 dist-test-slave-14jq krb5kdc[25857](info): commencing operation
krb5kdc: starting...
W20260927 15:25:54.054790 3033 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/test-user.keytab.
Sep 27 15:25:54 dist-test-slave-14jq krb5kdc[25857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:25:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
Entry for principal kudu/127.2.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab.
I20260927 15:25:54.382812 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42061
--webserver_interface=127.2.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:25:54.751811 25870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:54.752166 25870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:54.752231 25870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:54.752315 25870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:54.759761 25870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:54.759864 25870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:54.759992 25870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:54.760080 25870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:54.760182 25870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:54.771191 25870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.126.keytab
--principal=kudu/127.2.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.126:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:54.772616 25870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:54.773998 25870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:54.787391 25878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:54.787609 25875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:54.787529 25876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:54.789088 25870 server_base.cc:1034] running on GCE node
I20260927 15:25:54.790159 25870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:54.791742 25870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:54.793073 25870 hybrid_clock.cc:648] HybridClock initialized: now 1790522754792965 us; error 69 us; skew 500 ppm
I20260927 15:25:54.793488 25870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:54 dist-test-slave-14jq krb5kdc[25857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:54.797333 25870 init.cc:377] Logged in from keytab as kudu/127.2.246.126@KRBTEST.COM (short username kudu)
I20260927 15:25:54.799501 25870 webserver.cc:460] Webserver started at http://127.2.246.126:33885/ using document root <none> and password file <none>
I20260927 15:25:54.800349 25870 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:54.800455 25870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:54.800838 25870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:54.804628 25870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data/instance:
uuid: "aec8f7753e31441dbf072076c8cdf323"
format_stamp: "Formatted at 2026-09-27 15:25:54 on dist-test-slave-14jq"
server_key: "baf5f04ee3b36ab74a450216a719fe77"
server_key_iv: "3a40ae00f4e3939b492403297c728993"
server_key_version: "encryptionkey@0"
I20260927 15:25:54.805532 25870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance:
uuid: "aec8f7753e31441dbf072076c8cdf323"
format_stamp: "Formatted at 2026-09-27 15:25:54 on dist-test-slave-14jq"
server_key: "baf5f04ee3b36ab74a450216a719fe77"
server_key_iv: "3a40ae00f4e3939b492403297c728993"
server_key_version: "encryptionkey@0"
I20260927 15:25:54.811697 25870 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 15:25:54.816318 25885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:54.818346 25870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:25:54.818601 25870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal
uuid: "aec8f7753e31441dbf072076c8cdf323"
format_stamp: "Formatted at 2026-09-27 15:25:54 on dist-test-slave-14jq"
server_key: "baf5f04ee3b36ab74a450216a719fe77"
server_key_iv: "3a40ae00f4e3939b492403297c728993"
server_key_version: "encryptionkey@0"
I20260927 15:25:54.818877 25870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:54.841429 25870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:54.848237 25870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:54.848675 25870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:54.875433 25870 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.126:42061
I20260927 15:25:54.875439 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.126:42061 every 8 connection(s)
I20260927 15:25:54.877377 25870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/data/info.pb
I20260927 15:25:54.881963 25937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:54.886978 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25870
I20260927 15:25:54.887228 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-0/wal/instance
I20260927 15:25:54.887769 3033 external_mini_cluster.cc:1428] Setting key 90dfda64c999409d606f283c8d33d45d
I20260927 15:25:54.889060 25937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:54 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.125@KRBTEST.COM; defaulting to no policy
I20260927 15:25:54.918898 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.893220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42061 (local address 127.0.0.1:46146)
0927 15:25:54.894545 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:54.894591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:54.896501 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:54.897404 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:54.897411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:54.898220 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:54.898671 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:54.898700 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:54.900970 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:54.900977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:54.902513 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:54.902537 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:54.904212 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:54.904218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:54.904391 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:54.905320 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:54.905349 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:54.908638 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:54.912962 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:54.912985 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:54.913004 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:54.913865 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:54.914542 (+ 677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:54.914551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:54.914558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:54.914838 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:54.915935 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:54.915947 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:54.916283 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 15:25:54.916620 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 15:25:54.917481 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":186,"thread_start_us":887,"threads_started":1}
I20260927 15:25:54.918941 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.893196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.2.246.126:42061)
0927 15:25:54.894217 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:54.894240 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:54.894740 (+ 500us) server_negotiation.cc:407] Connection header received
0927 15:25:54.896908 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:54.896915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:54.897172 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:54.897381 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:54.898927 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:54.900709 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:54.902755 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:54.903874 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:54.904109 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:54.909060 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:54.909072 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:54.909092 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:54.909136 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:54.912699 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:54.914112 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:54.914124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:54.914129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:54.914375 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:54.914999 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:54.915003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:54.915005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:54.915742 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:54.915942 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:54.916815 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 15:25:54.917221 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":746,"thread_start_us":469,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.125@KRBTEST.COM" created.
I20260927 15:25:54.922969 25937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:54.924396 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.923675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40016)
0927 15:25:54.923987 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:54.924184 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 15:25:54.924764 25888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
I20260927 15:25:54.926882 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.926308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40030)
0927 15:25:54.926526 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:54.926649 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:25:54.927205 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:36423: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
Entry for principal kudu/127.2.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab.
I20260927 15:25:54.962731 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:36423
--webserver_interface=127.2.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:25:54.976729 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } attempt: 1
I20260927 15:25:54.978510 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.977786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40038)
0927 15:25:54.978160 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:54.978323 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 15:25:54.979889 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:54.979478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40042)
0927 15:25:54.979642 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:54.979732 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:25:54.980099 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:36423: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
I20260927 15:25:55.048441 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } attempt: 2
I20260927 15:25:55.050189 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.049478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40058)
0927 15:25:55.049808 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.049998 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 15:25:55.051659 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.051165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40074)
0927 15:25:55.051352 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.051477 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 15:25:55.051890 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:36423: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
I20260927 15:25:55.143280 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } attempt: 3
I20260927 15:25:55.144793 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.144256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40084)
0927 15:25:55.144502 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.144617 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 15:25:55.146010 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.145533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40092)
0927 15:25:55.145687 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.145777 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:25:55.146221 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:36423: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
I20260927 15:25:55.289546 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } attempt: 4
I20260927 15:25:55.291090 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.290477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40100)
0927 15:25:55.290834 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.290931 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 15:25:55.292141 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.291777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40110)
0927 15:25:55.291927 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.292018 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:25:55.292341 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.125:36423: Network error: Client connection negotiation failed: client connection to 127.2.246.125:36423: connect: Connection refused (error 111)
W20260927 15:25:55.307197 25944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:55.307591 25944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:55.307717 25944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:55.307865 25944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:55.315455 25944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:55.315595 25944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:55.315718 25944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:55.315809 25944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:55.315930 25944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:55.326495 25944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.125.keytab
--principal=kudu/127.2.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.125:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:55.327896 25944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:55.329252 25944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:55.342883 25952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:55.342954 25949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:55.342957 25950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:55.344163 25944 server_base.cc:1034] running on GCE node
I20260927 15:25:55.345252 25944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:55.347349 25944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:55.348659 25944 hybrid_clock.cc:648] HybridClock initialized: now 1790522755348559 us; error 76 us; skew 500 ppm
I20260927 15:25:55.349113 25944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:55 dist-test-slave-14jq krb5kdc[25857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:55.353215 25944 init.cc:377] Logged in from keytab as kudu/127.2.246.125@KRBTEST.COM (short username kudu)
I20260927 15:25:55.355155 25944 webserver.cc:460] Webserver started at http://127.2.246.125:36685/ using document root <none> and password file <none>
I20260927 15:25:55.356154 25944 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:55.356271 25944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:55.356606 25944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:55.360416 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data/instance:
uuid: "8b2070faf0124c37a53983e399d2b12b"
format_stamp: "Formatted at 2026-09-27 15:25:55 on dist-test-slave-14jq"
server_key: "058519b2bccb1975f14542de553a5144"
server_key_iv: "9ab9f8e423810fc987edc896bec7ddc1"
server_key_version: "encryptionkey@0"
I20260927 15:25:55.361436 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal/instance:
uuid: "8b2070faf0124c37a53983e399d2b12b"
format_stamp: "Formatted at 2026-09-27 15:25:55 on dist-test-slave-14jq"
server_key: "058519b2bccb1975f14542de553a5144"
server_key_iv: "9ab9f8e423810fc987edc896bec7ddc1"
server_key_version: "encryptionkey@0"
I20260927 15:25:55.368741 25944 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260927 15:25:55.374150 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:55.376307 25944 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:25:55.376592 25944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal
uuid: "8b2070faf0124c37a53983e399d2b12b"
format_stamp: "Formatted at 2026-09-27 15:25:55 on dist-test-slave-14jq"
server_key: "058519b2bccb1975f14542de553a5144"
server_key_iv: "9ab9f8e423810fc987edc896bec7ddc1"
server_key_version: "encryptionkey@0"
I20260927 15:25:55.376912 25944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:55.400604 25944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:55.405794 25944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:55.406215 25944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:55.429312 25944 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.125:36423
I20260927 15:25:55.429368 26010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.125:36423 every 8 connection(s)
I20260927 15:25:55.431090 25944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/data/info.pb
I20260927 15:25:55.435346 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:55.438021 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 25944
I20260927 15:25:55.438227 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-1/wal/instance
I20260927 15:25:55.438767 3033 external_mini_cluster.cc:1428] Setting key 2faf339896e1335fdb6f68f47f107b6e
I20260927 15:25:55.446355 26011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:55 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.246.124@KRBTEST.COM" created.
I20260927 15:25:55.475360 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.450918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.2.246.126:42061)
0927 15:25:55.451639 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:55.451648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:55.452697 (+ 1049us) server_negotiation.cc:407] Connection header received
0927 15:25:55.455582 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:55.455588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:55.455699 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:55.455849 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:55.458334 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.459257 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.461914 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.462492 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.462636 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.467681 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:55.467688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:55.467692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:55.467744 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:55.471195 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.472292 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.472302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.472305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.472437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.472978 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.472986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.472992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.473352 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:55.473549 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:55.474647 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0927 15:25:55.474930 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":335,"threads_started":1}
I20260927 15:25:55.476629 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.451578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42061 (local address 127.0.0.1:46158)
0927 15:25:55.452473 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.452529 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:55.455155 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:55.456177 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:55.456192 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:55.457094 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:55.457988 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.458016 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.459479 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.459487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.461591 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.461609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.462778 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.462787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.463134 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.464042 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:55.464078 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:55.467324 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:55.471435 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.471446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.471467 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.472137 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.472561 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.472566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.472570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.472780 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.473502 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:55.473513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:55.474187 (+ 674us) client_negotiation.cc:769] Sending connection context
0927 15:25:55.474517 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 15:25:55.475200 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260927 15:25:55.479470 26011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:55 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:25:55.496644 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.480441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40124)
0927 15:25:55.480834 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.480858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:55.481026 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:55.482985 (+ 1959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:55.482993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:55.483614 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:55.483868 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.483882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.485597 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.485606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.486515 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.486525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.487374 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.487380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.487547 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.488189 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:55.488207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:55.490773 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:55.494179 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.494184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.494188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.494546 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.494900 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.494906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.494910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.495044 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.495637 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:55.495646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:55.495786 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 15:25:55.495946 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:25:55.496234 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 15:25:55.496737 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.480855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.2.246.125:36423)
0927 15:25:55.481951 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:55.481962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:55.482003 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:25:55.482081 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:55.482092 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:55.482376 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:55.482567 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:55.484105 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.485440 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.486653 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.487202 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.487353 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.491059 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:55.491071 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:55.491085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:55.491112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:55.494035 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.494674 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.494683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.494686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.494796 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.495162 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.495166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.495170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.495518 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:55.495739 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:55.496075 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 15:25:55.496358 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":758,"thread_start_us":558,"threads_started":1}
I20260927 15:25:55.499182 26011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:55.500691 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.500160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49152)
0927 15:25:55.500421 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.500538 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 15:25:55.501019 25963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.502830 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.502480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49160)
0927 15:25:55.502637 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.502713 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:25:55.503077 26011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
Entry for principal kudu/127.2.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab.
I20260927 15:25:55.514166 3033 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
/tmp/dist-test-taskbUGd5v/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:37463
--webserver_interface=127.2.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.246.84:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:25:55.552451 26011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 1
I20260927 15:25:55.554180 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.553429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49168)
0927 15:25:55.553744 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.553988 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260927 15:25:55.555485 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.555008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49172)
0927 15:25:55.555197 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.555309 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:25:55.555706 26011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.591616 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } attempt: 5
Sep 27 15:25:55 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
I20260927 15:25:55.607337 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.592916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.2.246.125:36423)
0927 15:25:55.593170 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:55.593176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:55.593194 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:25:55.593304 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:55.593309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:55.593394 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:55.593552 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:55.594439 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.595119 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.596324 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.596785 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.596921 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.601097 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:55.601103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:55.601107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:55.601143 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:55.603941 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.605128 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.605133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.605135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.605259 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:55.605769 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:55.605774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:55.605776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:55.606165 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:55.606312 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:55.606852 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 15:25:55.607033 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:25:55.607522 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.592442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40140)
0927 15:25:55.592660 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.592684 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:55.592806 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:55.593588 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:55.593593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:55.594099 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:55.594281 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.594290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.595253 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.595260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.596157 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:55.596169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:55.596957 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:55.596963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:55.597179 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:55.597880 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:55.597919 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:55.600818 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:55.604186 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.604193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.604200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.604943 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.605395 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:55.605401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:55.605405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:55.605587 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:55.606309 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:55.606314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:55.606533 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 15:25:55.606719 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 15:25:55.607100 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:25:55.609035 25937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:55.610409 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.609905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49184)
0927 15:25:55.610127 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.610238 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:25:55.611539 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.611201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49196)
0927 15:25:55.611363 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.611448 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:25:55.611814 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.623955 26011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 2
I20260927 15:25:55.625626 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.624744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49198)
0927 15:25:55.625071 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.625359 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 15:25:55.627154 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.626620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49210)
0927 15:25:55.626820 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.626949 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:25:55.627550 26011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.663064 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 1
I20260927 15:25:55.664587 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.664015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49214)
0927 15:25:55.664268 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.664402 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 15:25:55.665699 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.665381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49226)
0927 15:25:55.665524 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.665601 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:25:55.665968 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.718920 26011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 3
I20260927 15:25:55.720528 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.719865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49234)
0927 15:25:55.720162 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.720340 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 15:25:55.722026 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.721482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49238)
0927 15:25:55.721733 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.721889 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260927 15:25:55.722234 26011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.734189 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 2
I20260927 15:25:55.735730 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.735123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49244)
0927 15:25:55.735427 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.735540 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 15:25:55.737100 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.736520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49254)
0927 15:25:55.736765 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.736893 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 15:25:55.737442 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.843755 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 3
I20260927 15:25:55.845347 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.844505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49268)
0927 15:25:55.844871 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.845058 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 15:25:55.846887 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.846214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49270)
0927 15:25:55.846515 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.846677 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260927 15:25:55.847255 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:55.865500 26011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 4
I20260927 15:25:55.867156 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.866534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49272)
0927 15:25:55.866864 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.866981 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 15:25:55.868417 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:55.868073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49284)
0927 15:25:55.868216 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:55.868316 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:25:55.868625 26011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
W20260927 15:25:55.950677 26019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:25:55.951231 26019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:25:55.951380 26019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:25:55.951562 26019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:25:55.960340 26019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:25:55.960572 26019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:25:55.960711 26019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:25:55.960821 26019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:25:55.960952 26019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:25:55.973734 26019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.84:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.246.126:42061,127.2.246.125:36423,127.2.246.124:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/krb5kdc/kudu_127.2.246.124.keytab
--principal=kudu/127.2.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.124:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:25:55.975623 26019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:25:55.977363 26019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:25:55.992225 26024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:55.992316 26025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:25:55.992211 26027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:25:55.998636 26019 server_base.cc:1034] running on GCE node
I20260927 15:25:55.999898 26019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:25:56.002134 26019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:25:56.003481 26019 hybrid_clock.cc:648] HybridClock initialized: now 1790522756003397 us; error 50 us; skew 500 ppm
I20260927 15:25:56.003989 26019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522756, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:25:56.013708 26019 init.cc:377] Logged in from keytab as kudu/127.2.246.124@KRBTEST.COM (short username kudu)
I20260927 15:25:56.016140 26019 webserver.cc:460] Webserver started at http://127.2.246.124:41713/ using document root <none> and password file <none>
I20260927 15:25:56.017119 26019 fs_manager.cc:359] Metadata directory not provided
I20260927 15:25:56.017243 26019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:25:56.017620 26019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:25:56.021596 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data/instance:
uuid: "6c3d880c0e2244018fe10b0af32dac43"
format_stamp: "Formatted at 2026-09-27 15:25:56 on dist-test-slave-14jq"
server_key: "d8e1f450cde20e68ac3df4db4d793e73"
server_key_iv: "a372a4a6a1673e69728e62132adc0172"
server_key_version: "encryptionkey@0"
I20260927 15:25:56.022838 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal/instance:
uuid: "6c3d880c0e2244018fe10b0af32dac43"
format_stamp: "Formatted at 2026-09-27 15:25:56 on dist-test-slave-14jq"
server_key: "d8e1f450cde20e68ac3df4db4d793e73"
server_key_iv: "a372a4a6a1673e69728e62132adc0172"
server_key_version: "encryptionkey@0"
I20260927 15:25:56.024529 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 4
I20260927 15:25:56.025893 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.025172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49296)
0927 15:25:56.025474 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.025627 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 15:25:56.027251 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.026702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49312)
0927 15:25:56.026913 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.027040 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 15:25:56.027556 25937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.246.124:37463: Network error: Client connection negotiation failed: client connection to 127.2.246.124:37463: connect: Connection refused (error 111)
I20260927 15:25:56.030743 26019 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 15:25:56.035629 26034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:25:56.037693 26019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:25:56.038038 26019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal
uuid: "6c3d880c0e2244018fe10b0af32dac43"
format_stamp: "Formatted at 2026-09-27 15:25:56 on dist-test-slave-14jq"
server_key: "d8e1f450cde20e68ac3df4db4d793e73"
server_key_iv: "a372a4a6a1673e69728e62132adc0172"
server_key_version: "encryptionkey@0"
I20260927 15:25:56.038378 26019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:25:56.070701 26019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:25:56.076308 26019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:25:56.076682 26019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:25:56.100788 26019 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.124:37463
I20260927 15:25:56.100812 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.246.124:37463 every 8 connection(s)
I20260927 15:25:56.102777 26019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/data/info.pb
I20260927 15:25:56.107033 26086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:25:56.108026 3033 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu as pid 26019
I20260927 15:25:56.108239 3033 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUGd5v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522351366233-3033-0/minicluster-data/master-2/wal/instance
I20260927 15:25:56.108793 3033 external_mini_cluster.cc:1428] Setting key f2cbde7ae7c824428617def167531459
I20260927 15:25:56.117801 26086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522756, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:56.137044 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.111193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.2.246.124:37463)
0927 15:25:56.112269 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.112284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.112346 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:25:56.114378 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.114409 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.114999 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.115460 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.117341 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.119047 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.120911 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.122265 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.122502 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.127432 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.127442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.127457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.127514 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.131837 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.132834 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.132843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.132853 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.133047 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.133465 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.133472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.133475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.134276 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.134628 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.135134 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.135713 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":494,"threads_started":1}
I20260927 15:25:56.138453 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.121161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.2.246.126:42061)
0927 15:25:56.121826 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.121872 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.122644 (+ 772us) server_negotiation.cc:407] Connection header received
0927 15:25:56.122894 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.122900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.123021 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.123187 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.124579 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.125321 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.126841 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.127246 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.127390 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.132014 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.132020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.132025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.132081 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.134949 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.136016 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.136028 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.136031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.136161 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.136588 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.136599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.136602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.136893 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.137059 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.137707 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.138000 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
I20260927 15:25:56.138495 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.121229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.126:42061 (local address 127.0.0.1:46162)
0927 15:25:56.122336 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.122445 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:56.122709 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:56.123284 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:56.123293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:56.124159 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:56.124409 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.124419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.125482 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.125489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.126673 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.126686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.127475 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.127481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.127679 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.128434 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:56.128464 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:56.131757 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:56.135140 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.135157 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.135166 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.135871 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.136287 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.136300 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.136305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.136467 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.137040 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:56.137049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:56.137355 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 15:25:56.137587 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 15:25:56.137999 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":580,"threads_started":1}
I20260927 15:25:56.140578 26086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522756, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.126@KRBTEST.COM
I20260927 15:25:56.159183 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.141283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.125:36423 (local address 127.0.0.1:40152)
0927 15:25:56.141420 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.141440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:56.141536 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:56.142424 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:56.142429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:56.142890 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:56.143078 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.143087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.144703 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.144710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.146330 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.146350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.147414 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.147421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.147669 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.148596 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:56.148629 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:56.152031 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:56.156160 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.156170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.156175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.156588 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.157071 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.157076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.157080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.157304 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.158108 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:56.158113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:56.158320 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 15:25:56.158501 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 15:25:56.158813 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 15:25:56.159443 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.141469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.2.246.125:36423)
0927 15:25:56.142063 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.142071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.142090 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:25:56.142150 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.142155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.142231 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.142370 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.143287 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.144426 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.146535 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.147186 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.147372 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.152398 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.152405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.152416 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.152465 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.155926 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.156727 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.156735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.156740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.156925 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.157470 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.157476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.157479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.157975 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.158141 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.158617 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.158902 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":289,"threads_started":1}
I20260927 15:25:56.159588 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.140988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.2.246.126:42061)
0927 15:25:56.141188 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.141193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.141285 (+ 92us) server_negotiation.cc:407] Connection header received
0927 15:25:56.141547 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.141554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.141683 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.141814 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.143517 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.144550 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.146544 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.147283 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.147399 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.153230 (+ 5831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.153235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.153238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.153275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.156069 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.157061 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.157066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.157070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.157252 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.157926 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.157930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.157932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.158321 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.158641 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.159036 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.159281 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:25:56.160876 26086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:25:56.167917 26011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 5
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522756, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.124@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.246.125@KRBTEST.COM
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.125@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:25:56.179858 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.161793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49334)
0927 15:25:56.162321 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.162342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:56.162469 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:56.162899 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:56.162903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:56.163337 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:56.163500 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.163509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.164927 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.164932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.166318 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.166335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.167212 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.167217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.167397 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.168237 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:56.168268 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:56.171908 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:56.176678 (+ 4770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.176685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.176690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.177108 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.177526 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.177530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.177532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.177711 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.178611 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:56.178617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:56.178813 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 15:25:56.178996 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 15:25:56.179402 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410}
I20260927 15:25:56.180536 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.161793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.2.246.124:37463)
0927 15:25:56.162030 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.162036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.162449 (+ 413us) server_negotiation.cc:407] Connection header received
0927 15:25:56.162624 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.162629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.162708 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.162840 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.163717 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.164722 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.166490 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.167024 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.167179 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.172132 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.172139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.172144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.172187 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.176512 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.177245 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.177252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.177260 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.177375 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.177944 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.177954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.177961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.178451 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.178642 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.179161 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.179650 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":107}
I20260927 15:25:56.181524 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.162121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.2.246.125:36423)
0927 15:25:56.162301 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.162307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.162326 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:25:56.162389 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.162397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.162480 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.162582 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.164264 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.165185 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.166864 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.167518 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.167634 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.173168 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.173173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.173177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.173221 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.177299 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.178380 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.178385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.178388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.178534 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.179128 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.179133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.179136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.179579 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.179854 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.181011 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.181261 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:25:56.188125 3033 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:25:56.188304 26086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: Bootstrap starting.
I20260927 15:25:56.188421 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.168584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49338)
0927 15:25:56.168867 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.168891 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:56.169007 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:56.170000 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:56.170005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:56.170598 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:56.170860 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.170874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.172523 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.172529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.173674 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.173687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.174849 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.174857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.175612 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.176447 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:56.176478 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:56.179708 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:56.185347 (+ 5639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.185355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.185362 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.185797 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.186234 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.186239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.186242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.186374 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.187152 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:56.187158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:56.187396 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 15:25:56.187598 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:25:56.187987 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 15:25:56.188714 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.168753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.2.246.124:37463)
0927 15:25:56.169676 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.169680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.169698 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:25:56.169737 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.169740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.169797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.169983 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.171097 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.172349 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.173867 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.174642 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.174839 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.180031 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.180040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.180047 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.180091 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.185189 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.185953 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.185960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.185964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.186111 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.186485 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.186491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.186494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.186937 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.187086 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.187729 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.188206 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":600,"threads_started":1}
I20260927 15:25:56.193325 26086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:56.194005 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: Bootstrap starting.
I20260927 15:25:56.195034 26086 log.cc:824] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:56.191752 25912 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aec8f7753e31441dbf072076c8cdf323"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46166
I20260927 15:25:56.197700 26011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:56.198824 26011 log.cc:824] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:56.199352 26086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: No bootstrap required, opened a new log
I20260927 15:25:56.201628 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: No bootstrap required, opened a new log
I20260927 15:25:56.208900 26011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.209214 26011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2070faf0124c37a53983e399d2b12b, State: Initialized, Role: FOLLOWER
I20260927 15:25:56.209723 26011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.210780 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.211182 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.211503 26086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.211992 26011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:56.212039 26086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3d880c0e2244018fe10b0af32dac43, State: Initialized, Role: FOLLOWER
I20260927 15:25:56.212620 26086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.214090 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.214950 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.215551 26086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:25:56.223248 26109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:25:56.223414 26109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:25:56.220899 25944 master_runner.cc:186] Error getting master registration for 127.2.246.126:42061: OK, instance_id { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" instance_seqno: 1790522754867222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:25:56.233206 25944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:25:56.236487 26120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:25:56.236663 26120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:25:56.230927 26019 master_runner.cc:186] Error getting master registration for 127.2.246.126:42061: OK, instance_id { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" instance_seqno: 1790522754867222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:25:56.244271 26019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:25:56.304773 25937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } attempt: 5
Sep 27 15:25:56 dist-test-slave-14jq krb5kdc[25857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.246.126@KRBTEST.COM for kudu/127.2.246.124@KRBTEST.COM
I20260927 15:25:56.325169 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.305730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.124:37463 (local address 127.0.0.1:49352)
0927 15:25:56.306044 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:25:56.306073 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:25:56.306243 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:25:56.307081 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:25:56.307088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:25:56.307753 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:25:56.308031 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.308046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.309601 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.309607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.311292 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:25:56.311309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.312445 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.312452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:25:56.312667 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.313522 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:25:56.313552 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:25:56.316881 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:25:56.321762 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.321769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.321774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.322445 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.322985 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:25:56.322990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:25:56.322994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:25:56.323186 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:25:56.324006 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:25:56.324011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:25:56.324185 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 15:25:56.324330 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:25:56.324677 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 15:25:56.326218 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.306405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.2.246.124:37463)
0927 15:25:56.306637 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.306645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.306667 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:25:56.306732 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.306738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.306860 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.306971 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.308256 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.309421 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.311482 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.312234 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.312409 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.317209 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.317217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.317223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.317278 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.321534 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.322609 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.322614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.322618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.322823 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.323328 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.323333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.323337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.323705 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.323943 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.325560 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.325861 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:25:56.330653 26040 tablet.cc:2369] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:56.334686 25937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323: Bootstrap starting.
I20260927 15:25:56.338971 25937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323: Neither blocks nor log segments found. Creating new log.
I20260927 15:25:56.340428 25937 log.cc:824] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323: Log is configured to *not* fsync() on all Append() calls
I20260927 15:25:56.345929 25937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323: No bootstrap required, opened a new log
I20260927 15:25:56.358400 25937 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.359157 25937 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec8f7753e31441dbf072076c8cdf323, State: Initialized, Role: FOLLOWER
I20260927 15:25:56.360059 25937 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.361732 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.362308 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.364356 25912 raft_consensus.cc:482] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:25:56.364575 25912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:56.366017 25937 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:25:56.368326 25912 raft_consensus.cc:504] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.371160 25986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aec8f7753e31441dbf072076c8cdf323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b2070faf0124c37a53983e399d2b12b"
I20260927 15:25:56.371673 25986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:56.375459 25986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec8f7753e31441dbf072076c8cdf323 in term 1.
I20260927 15:25:56.376478 25888 leader_election.cc:302] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2070faf0124c37a53983e399d2b12b, aec8f7753e31441dbf072076c8cdf323; no voters:
I20260927 15:25:56.377151 26123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:25:56.378548 26061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aec8f7753e31441dbf072076c8cdf323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c3d880c0e2244018fe10b0af32dac43"
I20260927 15:25:56.379102 26061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:25:56.383368 26061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec8f7753e31441dbf072076c8cdf323 in term 1.
I20260927 15:25:56.384294 25912 leader_election.cc:288] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [CANDIDATE]: Term 1 election: Requested vote from peers 8b2070faf0124c37a53983e399d2b12b (127.2.246.125:36423), 6c3d880c0e2244018fe10b0af32dac43 (127.2.246.124:37463)
I20260927 15:25:56.386241 26123 raft_consensus.cc:686] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [term 1 LEADER]: Becoming Leader. State: Replica: aec8f7753e31441dbf072076c8cdf323, State: Running, Role: LEADER
I20260927 15:25:56.387075 26123 consensus_queue.cc:237] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } }
I20260927 15:25:56.395988 26125 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec8f7753e31441dbf072076c8cdf323. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.396832 26125 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: This master's current role is: LEADER
W20260927 15:25:56.409760 25965 tablet.cc:2369] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:25:56.415366 26129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:25:56.450873 26129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:25:56.458192 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.424769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.2.246.125:36423)
0927 15:25:56.425066 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.425076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.438464 (+ 13388us) server_negotiation.cc:407] Connection header received
0927 15:25:56.438752 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.438757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.438851 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.438932 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.441160 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.442555 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.444636 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.445428 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.445536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.449244 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.449251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.449255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.449304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.454062 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.454984 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.454990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.454996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.455157 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.455623 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.455628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.455631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.456080 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.456388 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.457427 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.457682 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 15:25:56.461084 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.414621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.2.246.124:37463)
0927 15:25:56.414827 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.414833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.424786 (+ 9953us) server_negotiation.cc:407] Connection header received
0927 15:25:56.425747 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.425754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.425899 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.425975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.433994 (+ 8019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.435158 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.440255 (+ 5097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.441013 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.441138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.447753 (+ 6615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.447764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.447773 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.447844 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.457553 (+ 9709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.458591 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.458596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.458599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.458766 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.459217 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.459223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.459225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.459580 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.459895 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.460449 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.460692 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:25:56.466203 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:56.410745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.2.246.126:42061)
0927 15:25:56.410942 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.410950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.420013 (+ 9063us) server_negotiation.cc:407] Connection header received
0927 15:25:56.421230 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.421237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.421332 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.421411 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.429180 (+ 7769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.430409 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.437509 (+ 7100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:56.438320 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:56.438444 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:56.446616 (+ 8172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:25:56.446627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:25:56.446635 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:25:56.446733 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:25:56.462218 (+ 15485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.463166 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.463172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.463177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.463365 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:25:56.463807 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:25:56.463812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:25:56.463816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:25:56.464210 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:25:56.464467 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:56.465312 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 15:25:56.465660 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:25:56.478018 25986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [term 1 FOLLOWER]: Refusing update from remote peer aec8f7753e31441dbf072076c8cdf323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:56.478021 26061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [term 1 FOLLOWER]: Refusing update from remote peer aec8f7753e31441dbf072076c8cdf323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:25:56.479453 26125 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:56.480343 26123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:25:56.502631 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec8f7753e31441dbf072076c8cdf323. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.503103 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.514485 26141 mvcc.cc:205] Tried to move back new op lower bound from 7333981210518593536 to 7333981210169114624. Current Snapshot: MvccSnapshot[applied={T|T < 7333981210518593536}]
I20260927 15:25:56.518913 26125 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.519308 26125 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:56.519718 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec8f7753e31441dbf072076c8cdf323. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.519974 26145 mvcc.cc:205] Tried to move back new op lower bound from 7333981210518593536 to 7333981210169114624. Current Snapshot: MvccSnapshot[applied={T|T < 7333981210518593536}]
I20260927 15:25:56.520219 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.522768 26129 catalog_manager.cc:1347] Generated new cluster ID: 5a0b1e3f7e3a4d2d91e8d40b04bd3736
I20260927 15:25:56.522918 26129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:25:56.522831 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.523159 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.539036 26143 mvcc.cc:205] Tried to move back new op lower bound from 7333981210518593536 to 7333981210169114624. Current Snapshot: MvccSnapshot[applied={T|T < 7333981210518593536}]
I20260927 15:25:56.540776 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.541747 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323 [sys.catalog]: This master's current role is: LEADER
I20260927 15:25:56.550293 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aec8f7753e31441dbf072076c8cdf323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8f7753e31441dbf072076c8cdf323" member_type: VOTER last_known_addr { host: "127.2.246.126" port: 42061 } } peers { permanent_uuid: "8b2070faf0124c37a53983e399d2b12b" member_type: VOTER last_known_addr { host: "127.2.246.125" port: 36423 } } peers { permanent_uuid: "6c3d880c0e2244018fe10b0af32dac43" member_type: VOTER last_known_addr { host: "127.2.246.124" port: 37463 } } }
I20260927 15:25:56.550616 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:25:56.567277 26154 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:25:56.598155 26129 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:25:56.600165 26129 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:25:56.625433 26129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aec8f7753e31441dbf072076c8cdf323: Generated new TSK 0
I20260927 15:25:56.626926 26129 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:25:56.796727 26090 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.782543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.2.246.126:42061)
0927 15:25:56.782863 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.782872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.792734 (+ 9862us) server_negotiation.cc:407] Connection header received
0927 15:25:56.794609 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.794632 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.794764 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.794863 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.796370 (+ 1507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46188: BlockingRecv error: recv got EOF from 127.0.0.1:46188 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 15:25:56.804481 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.799075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.2.246.125:36423)
0927 15:25:56.799365 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.799373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.800476 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 15:25:56.802481 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.802488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.802598 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.802702 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.804113 (+ 1411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40176: BlockingRecv error: recv got EOF from 127.0.0.1:40176 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 15:25:56.806527 26093 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.787222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.2.246.124:37463)
0927 15:25:56.787487 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.787498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.798195 (+ 10697us) server_negotiation.cc:407] Connection header received
0927 15:25:56.801145 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.801153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.801264 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.801359 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.806213 (+ 4854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49364: BlockingRecv error: recv got EOF from 127.0.0.1:49364 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 15:25:56.857445 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.841908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.2.246.125:36423)
0927 15:25:56.842255 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.842267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.854762 (+ 12495us) server_negotiation.cc:407] Connection header received
0927 15:25:56.855113 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.855121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.855326 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.855467 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.857079 (+ 1612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40188: BlockingRecv error: recv got EOF from 127.0.0.1:40188 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 15:25:56.865161 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.862634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.2.246.125:36423)
0927 15:25:56.862896 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.862903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.863292 (+ 389us) server_negotiation.cc:407] Connection header received
0927 15:25:56.863544 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.863553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.863695 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.863884 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.864841 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40194: BlockingRecv error: recv got EOF from 127.0.0.1:40194 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260927 15:25:56.875713 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.872717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.2.246.125:36423)
0927 15:25:56.872961 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.872969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.873704 (+ 735us) server_negotiation.cc:407] Connection header received
0927 15:25:56.874048 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.874055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.874174 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.874309 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.875426 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40196: BlockingRecv error: recv got EOF from 127.0.0.1:40196 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 15:25:56.926232 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.923707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.2.246.125:36423)
0927 15:25:56.924027 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.924037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.924081 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:25:56.924416 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.924426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.924553 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.924681 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.925983 (+ 1302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40210: BlockingRecv error: recv got EOF from 127.0.0.1:40210 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260927 15:25:56.943598 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.940232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.2.246.125:36423)
0927 15:25:56.940516 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.940527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.941273 (+ 746us) server_negotiation.cc:407] Connection header received
0927 15:25:56.941572 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.941577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.941710 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.941849 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.943197 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40220: BlockingRecv error: recv got EOF from 127.0.0.1:40220 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 15:25:56.968441 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:56.966158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.2.246.125:36423)
0927 15:25:56.966416 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:56.966423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:56.966792 (+ 369us) server_negotiation.cc:407] Connection header received
0927 15:25:56.966965 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:56.966972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:56.967083 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:56.967250 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:56.968171 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40232: BlockingRecv error: recv got EOF from 127.0.0.1:40232 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 15:25:57.007771 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:57.005353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.2.246.125:36423)
0927 15:25:57.005613 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.005619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.006055 (+ 436us) server_negotiation.cc:407] Connection header received
0927 15:25:57.006275 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.006282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.006439 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.006607 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:57.007518 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40244: BlockingRecv error: recv got EOF from 127.0.0.1:40244 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 15:25:57.078469 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:57.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.2.246.125:36423)
0927 15:25:57.076463 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.076471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.076795 (+ 324us) server_negotiation.cc:407] Connection header received
0927 15:25:57.077003 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.077009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.077143 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.077296 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:57.078217 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40250: BlockingRecv error: recv got EOF from 127.0.0.1:40250 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260927 15:25:57.213230 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:25:57.210662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.2.246.125:36423)
0927 15:25:57.211028 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.211036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.211423 (+ 387us) server_negotiation.cc:407] Connection header received
0927 15:25:57.211705 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.211710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.211801 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.211936 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:25:57.212869 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40252: BlockingRecv error: recv got EOF from 127.0.0.1:40252 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 15:25:57.225697 26109 catalog_manager.cc:1259] Loaded cluster ID: 5a0b1e3f7e3a4d2d91e8d40b04bd3736
I20260927 15:25:57.225878 26109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: loading cluster ID for follower catalog manager: success
I20260927 15:25:57.228523 26109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: acquiring CA information for follower catalog manager: success
I20260927 15:25:57.230703 26109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b2070faf0124c37a53983e399d2b12b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:25:57.239251 26120 catalog_manager.cc:1259] Loaded cluster ID: 5a0b1e3f7e3a4d2d91e8d40b04bd3736
I20260927 15:25:57.239382 26120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: loading cluster ID for follower catalog manager: success
I20260927 15:25:57.242445 26120 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: acquiring CA information for follower catalog manager: success
I20260927 15:25:57.244987 26120 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c3d880c0e2244018fe10b0af32dac43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:25:57.480638 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:57.474047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.2.246.125:36423)
0927 15:25:57.474384 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.474392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.474680 (+ 288us) server_negotiation.cc:407] Connection header received
0927 15:25:57.474928 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.474933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.475140 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.475267 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:57.475770 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.476598 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.478160 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.478578 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.478704 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:57.479254 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:57.479864 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:57.480002 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:57.480190 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 15:25:57.480268 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 15:25:57.495625 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:57.488557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.2.246.124:37463)
0927 15:25:57.489254 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.489263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.489287 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:25:57.489583 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.489588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.489789 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.490056 (+ 267us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:57.490537 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.491531 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.492955 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.493411 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.493560 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:57.494201 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:57.494889 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:57.495079 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:57.495137 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:25:57.495240 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":323,"threads_started":1}
I20260927 15:25:57.514192 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:57.504649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.2.246.125:36423)
0927 15:25:57.504917 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.504926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.507198 (+ 2272us) server_negotiation.cc:407] Connection header received
0927 15:25:57.507405 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.507409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.507485 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.507599 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:57.508495 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.509633 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.511843 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.512595 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.512756 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:57.513256 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:57.513438 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:57.513527 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:57.513712 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:25:57.513811 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:25:57.514818 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:57.503933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.2.246.124:37463)
0927 15:25:57.504246 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.504256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.506950 (+ 2694us) server_negotiation.cc:407] Connection header received
0927 15:25:57.507153 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.507157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.507243 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.507405 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:57.508270 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.509538 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.512080 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.512651 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.512805 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:57.513731 (+ 926us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:57.513909 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:57.514017 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:57.514356 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:25:57.514476 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:25:57.515385 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:25:57.503476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.2.246.126:42061)
0927 15:25:57.506485 (+ 3009us) server_negotiation.cc:206] Beginning negotiation
0927 15:25:57.506495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:25:57.506529 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:25:57.506751 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:25:57.506761 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:25:57.507679 (+ 918us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:25:57.507927 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:25:57.508597 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.509347 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.511936 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:25:57.512700 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:25:57.512820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:25:57.514084 (+ 1264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:25:57.514614 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:25:57.514718 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:25:57.514922 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 15:25:57.515035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2801,"thread_start_us":2651,"threads_started":1}
I20260927 15:25:57.521358 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25870
I20260927 15:25:57.522725 25932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:57.654189 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25870
I20260927 15:25:57.678436 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 25944
I20260927 15:25:57.680153 26006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:57.812386 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 25944
I20260927 15:25:57.834489 3033 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu pid 26019
I20260927 15:25:57.835958 26081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:25:57.948851 3033 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUGd5v/build/asan/bin/kudu with pid 26019
2026-09-27T15:25:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5093 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10132 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (406516 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST